2024-03-25T17:51:00.928 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T17:51:00.928 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310 --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310/orig.config.yaml 2024-03-25T17:51:00.971 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:51:01.080 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621310' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIi43x2E99LNSF/3BhT5R/K4DVqzzb5IcZBBtUyCOAS2Q2zD3i8ZiImheSbixMlHx8DOIpUst3eNJrQHI7tnX70= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0PwWWJMGrLoeOuDecx/Tzk2ixaznhC4GV2yeyjfgWbXxDkUGy7GYa1rGtCFqD+fTjGcat1iHyLvQ8iavV7mGw= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T17:51:01.081 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T17:51:01.081 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T17:51:01.081 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T17:51:01.083 INFO:teuthology.task.internal:Checking packages... 2024-03-25T17:51:01.104 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T17:51:01.104 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T17:51:01.104 INFO:teuthology.packaging:ref: None 2024-03-25T17:51:01.104 INFO:teuthology.packaging:tag: None 2024-03-25T17:51:01.104 INFO:teuthology.packaging:branch: main 2024-03-25T17:51:01.104 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:51:01.104 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T17:51:01.355 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T17:51:01.356 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T17:51:01.366 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T17:51:01.367 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T17:51:01.374 INFO:teuthology.task.internal:Saving configuration 2024-03-25T17:51:01.390 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T17:51:01.396 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:51:01.421 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:11.947994', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIi43x2E99LNSF/3BhT5R/K4DVqzzb5IcZBBtUyCOAS2Q2zD3i8ZiImheSbixMlHx8DOIpUst3eNJrQHI7tnX70='} 2024-03-25T17:51:01.443 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:11.949726', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0PwWWJMGrLoeOuDecx/Tzk2ixaznhC4GV2yeyjfgWbXxDkUGy7GYa1rGtCFqD+fTjGcat1iHyLvQ8iavV7mGw='} 2024-03-25T17:51:01.443 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T17:51:01.449 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T17:51:01.449 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T17:51:01.450 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T17:51:01.518 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa8f7ac50d0>, signals=[15]) 2024-03-25T17:51:01.519 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T17:51:01.525 INFO:teuthology.task.internal:Opening connections... 2024-03-25T17:51:01.526 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2024-03-25T17:51:01.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:51:01.631 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-03-25T17:51:01.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:51:01.720 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T17:51:01.730 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2024-03-25T17:51:01.748 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2024-03-25T17:51:01.748 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2024-03-25T17:51:01.805 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream" 2024-03-25T17:51:01.805 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="9" 2024-03-25T17:51:01.805 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2024-03-25T17:51:01.805 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:51:01.806 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="9" 2024-03-25T17:51:01.806 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:51:01.806 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:51:01.806 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2024-03-25T17:51:01.806 INFO:teuthology.orchestra.run.smithi019.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:51:01.806 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:51:01.806 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:51:01.806 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:51:01.806 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:51:01.806 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:51:01.807 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2024-03-25T17:51:01.831 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-03-25T17:51:01.848 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-03-25T17:51:01.848 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-03-25T17:51:01.905 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2024-03-25T17:51:01.905 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="9" 2024-03-25T17:51:01.905 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2024-03-25T17:51:01.906 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:51:01.906 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="9" 2024-03-25T17:51:01.906 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:51:01.906 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:51:01.906 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-03-25T17:51:01.906 INFO:teuthology.orchestra.run.smithi188.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:51:01.906 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:51:01.906 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:51:01.906 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:51:01.906 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:51:01.906 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:51:01.907 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-03-25T17:51:01.922 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T17:51:01.936 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T17:51:01.949 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T17:51:01.949 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:51:01.952 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:51:01.968 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T17:51:01.979 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T17:51:01.979 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:51:02.009 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:51:02.032 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T17:51:02.089 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T17:51:02.105 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T17:51:02.105 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T17:51:02.106 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:51:02.106 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:51:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:02.123 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-03-25T17:51:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:02.124 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-03-25T17:51:02.180 INFO:teuthology.orchestra.run.smithi188.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:51:02.180 INFO:teuthology.task.kernel:Running kernel on smithi188: 5.14.0-361.el9.x86_64 2024-03-25T17:51:02.181 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum install -y kernel 2024-03-25T17:51:02.183 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:51:02.183 INFO:teuthology.task.kernel:Running kernel on smithi019: 5.14.0-361.el9.x86_64 2024-03-25T17:51:02.183 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum install -y kernel 2024-03-25T17:51:04.336 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T17:51:04.396 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T17:51:05.854 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-25T17:51:05.953 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-25T17:51:13.453 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.0 MB 00:07 2024-03-25T17:51:13.726 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.0 MB 00:07 2024-03-25T17:51:22.405 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 19 MB 00:07 2024-03-25T17:51:23.149 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 2.2 MB/s | 19 MB 00:08 2024-03-25T17:51:34.066 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - CRB 803 kB/s | 6.1 MB 00:07 2024-03-25T17:51:34.221 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 839 kB/s | 6.1 MB 00:07 2024-03-25T17:51:35.495 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-03-25T17:51:35.667 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-25T17:51:37.212 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-25T17:51:37.282 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T17:51:42.154 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-25T17:51:42.335 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-25T17:51:44.228 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:51:44.228 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:51:44.311 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T17:51:44.322 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T17:51:44.322 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-03-25T17:51:44.323 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T17:51:44.323 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T17:51:44.323 INFO:teuthology.orchestra.run.smithi188.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:51:44.323 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-03-25T17:51:44.323 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:51:44.323 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:51:44.323 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:51:44.323 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T17:51:44.323 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T17:51:44.323 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T17:51:44.323 INFO:teuthology.orchestra.run.smithi188.stdout:Install 4 Packages 2024-03-25T17:51:44.323 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T17:51:44.324 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 76 M 2024-03-25T17:51:44.324 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 123 M 2024-03-25T17:51:44.324 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T17:51:44.533 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:51:44.534 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:51:44.632 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T17:51:44.639 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T17:51:44.639 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-03-25T17:51:44.639 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout:Install 4 Packages 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 76 M 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 123 M 2024-03-25T17:51:44.640 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T17:51:44.654 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:44.656 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:44.656 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:44.760 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:44.761 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:44.761 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:44.804 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:44.815 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:44.838 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:44.873 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:44.880 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:44.922 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:44.941 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:44.952 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:44.976 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:45.010 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:45.018 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:45.031 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:45.044 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:45.044 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:45.165 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:45.191 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:45.212 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:45.363 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:45.364 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:45.382 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:45.382 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:45.383 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:45.418 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:45.419 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:45.453 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:45.454 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:45.490 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:45.522 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:45.536 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:45.553 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:45.568 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:45.615 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:45.652 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:45.721 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:45.841 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:45.872 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:45.881 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:45.895 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:45.926 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:45.927 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:45.990 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:45.995 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:45.999 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:46.016 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:46.026 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:46.035 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:46.036 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:46.041 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:46.057 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:46.067 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:46.073 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:46.075 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:46.111 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:46.133 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:46.133 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:46.138 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:46.153 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:46.159 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:46.160 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:46.184 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:46.184 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:46.282 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:46.283 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:46.297 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:46.302 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:46.302 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:46.317 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:46.462 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:46.462 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:46.478 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:46.629 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:46.652 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:46.676 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:46.811 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:46.836 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:46.845 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:46.859 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:46.863 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:46.870 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:46.897 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:46.904 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:46.937 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:46.938 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:46.950 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:46.972 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:46.977 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:46.990 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:46.991 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:46.995 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:47.005 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:47.010 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:47.018 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:47.055 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:47.068 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:47.073 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:47.073 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:47.093 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:47.094 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:47.114 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:47.116 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:47.116 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:47.133 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:47.141 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:47.175 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:47.177 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:47.185 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:47.185 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:47.194 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:47.195 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:47.202 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:47.222 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:47.223 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:47.238 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:47.240 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:47.260 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:47.300 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:47.300 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:47.300 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:47.327 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:47.336 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:47.337 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:47.376 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:47.402 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:47.413 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:47.451 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:47.464 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:47.487 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:47.489 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:47.499 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:47.525 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:47.532 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:47.540 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:47.561 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:47.600 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:47.617 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:47.637 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:47.673 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:47.687 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:47.710 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:47.750 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:47.750 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:47.784 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:47.794 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:47.819 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:47.826 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:47.826 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:47.879 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:47.897 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:47.904 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:47.913 INFO:teuthology.orchestra.run.smithi188.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 20 kB/s | 66 kB 00:03 2024-03-25T17:51:47.963 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:47.972 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:48.008 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:48.020 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:48.036 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:48.140 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:48.144 INFO:teuthology.orchestra.run.smithi019.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 19 kB/s | 66 kB 00:03 2024-03-25T17:51:48.306 INFO:teuthology.orchestra.run.smithi188.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 4.0 MB/s | 15 MB 00:03 2024-03-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi019.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.6 MB/s | 15 MB 00:04 2024-03-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi188.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 7.2 MB/s | 33 MB 00:04 2024-03-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi019.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 6.9 MB/s | 33 MB 00:04 2024-03-25T17:51:49.646 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:49.766 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:49.831 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:49.854 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:49.872 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:49.894 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:49.906 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:50.064 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:50.123 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:50.220 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:50.277 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:50.424 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:50.514 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:50.600 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:50.712 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:50.943 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:50.962 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:50.996 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:51.056 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:51.069 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:51.077 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:51.126 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:51.181 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:51.222 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:51.289 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:51.312 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:51.326 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:51.335 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:51.402 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:51.492 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:51.522 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:51.681 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:51.734 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:51.866 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:51.888 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:51.990 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:52.046 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:53.072 INFO:teuthology.orchestra.run.smithi188.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 5.3 MB/s | 27 MB 00:05 2024-03-25T17:51:53.077 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:53.077 INFO:teuthology.orchestra.run.smithi188.stdout:Total 8.7 MB/s | 76 MB 00:08 2024-03-25T17:51:53.311 INFO:teuthology.orchestra.run.smithi019.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 5.3 MB/s | 27 MB 00:05 2024-03-25T17:51:53.317 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:53.317 INFO:teuthology.orchestra.run.smithi019.stdout:Total 8.8 MB/s | 76 MB 00:08 2024-03-25T17:51:53.745 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T17:51:54.010 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T17:51:54.456 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T17:51:54.456 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T17:51:55.136 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T17:51:55.136 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T17:51:55.498 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T17:51:55.498 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T17:51:57.605 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T17:51:58.096 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T17:51:58.222 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:51:58.316 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:51:58.719 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:51:58.952 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:01.071 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:01.210 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:01.826 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:01.962 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:05.922 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:06.027 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:07.036 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:07.079 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:10.751 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:12.180 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:39.859 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:39.859 INFO:teuthology.orchestra.run.smithi188.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:52:39.859 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T17:52:39.905 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:40.306 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:40.307 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:40.307 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:40.307 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:40.759 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:40.759 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T17:52:40.759 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T17:52:40.759 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:40.760 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:52:40.760 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:52:40.760 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:52:40.760 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T17:52:40.760 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T17:52:41.076 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:52:41.178 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:41.178 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:52:41.179 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:52:41.179 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:52:41.179 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:52:41.179 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T17:52:41.179 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi188.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:52:41.179 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:52:41.258 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:41.259 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:52:41.260 DEBUG:teuthology.orchestra.run.smithi188:> sudo rpm -qi grub2-tools 2024-03-25T17:52:41.294 INFO:teuthology.orchestra.run.smithi188.stdout:Name : grub2-tools 2024-03-25T17:52:41.294 INFO:teuthology.orchestra.run.smithi188.stdout:Epoch : 1 2024-03-25T17:52:41.294 INFO:teuthology.orchestra.run.smithi188.stdout:Version : 2.06 2024-03-25T17:52:41.294 INFO:teuthology.orchestra.run.smithi188.stdout:Release : 68.el9 2024-03-25T17:52:41.294 INFO:teuthology.orchestra.run.smithi188.stdout:Architecture: x86_64 2024-03-25T17:52:41.294 INFO:teuthology.orchestra.run.smithi188.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:52:41.294 INFO:teuthology.orchestra.run.smithi188.stdout:Group : Unspecified 2024-03-25T17:52:41.294 INFO:teuthology.orchestra.run.smithi188.stdout:Size : 8264254 2024-03-25T17:52:41.294 INFO:teuthology.orchestra.run.smithi188.stdout:License : GPLv3+ 2024-03-25T17:52:41.294 INFO:teuthology.orchestra.run.smithi188.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:Packager : builder@centos.org 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:Vendor : CentOS 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:Summary : Support tools for GRUB. 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:Description : 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:hardware devices. 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T17:52:41.295 INFO:teuthology.orchestra.run.smithi188.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:52:41.296 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:52:41.296 INFO:teuthology.task.kernel:Updating grub on smithi188 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:52:41.296 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:52:41.922 INFO:teuthology.orchestra.run.smithi188.stderr:Generating grub configuration file ... 2024-03-25T17:52:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:42.987 INFO:teuthology.orchestra.run.smithi019.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:52:42.987 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T17:52:43.032 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:43.350 INFO:teuthology.orchestra.run.smithi188.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:52:43.387 INFO:teuthology.orchestra.run.smithi188.stderr:done 2024-03-25T17:52:43.390 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-03-25T17:52:43.405 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.qvet1vJBlx 2024-03-25T17:52:43.405 DEBUG:teuthology.orchestra.run.smithi188:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qvet1vJBlx 2024-03-25T17:52:43.411 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:43.411 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:43.411 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:43.411 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:43.475 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.qvet1vJBlx 2024-03-25T17:52:43.611 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.qvet1vJBlx is 6KB 2024-03-25T17:52:43.660 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.qvet1vJBlx 2024-03-25T17:52:43.673 DEBUG:teuthology.orchestra.run.smithi188:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:52:43.739 INFO:teuthology.orchestra.run.smithi188.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:52:43.739 INFO:teuthology.orchestra.run.smithi188.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:52:43.739 INFO:teuthology.orchestra.run.smithi188.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:52:43.739 INFO:teuthology.orchestra.run.smithi188.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:52:43.740 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:52:43.782 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:43.782 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T17:52:43.782 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T17:52:43.782 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:43.782 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:52:43.782 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:52:43.782 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:52:43.782 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T17:52:43.782 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T17:52:43.890 DEBUG:teuthology.orchestra.run.smithi188:> sudo shutdown -r now 2024-03-25T17:52:44.094 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:52:44.192 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:44.193 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:52:44.193 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:52:44.193 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:52:44.193 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:52:44.193 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T17:52:44.193 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi019.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:52:44.193 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:52:44.277 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:44.277 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:52:44.277 DEBUG:teuthology.orchestra.run.smithi019:> sudo rpm -qi grub2-tools 2024-03-25T17:52:44.354 INFO:teuthology.orchestra.run.smithi019.stdout:Name : grub2-tools 2024-03-25T17:52:44.354 INFO:teuthology.orchestra.run.smithi019.stdout:Epoch : 1 2024-03-25T17:52:44.354 INFO:teuthology.orchestra.run.smithi019.stdout:Version : 2.06 2024-03-25T17:52:44.354 INFO:teuthology.orchestra.run.smithi019.stdout:Release : 68.el9 2024-03-25T17:52:44.354 INFO:teuthology.orchestra.run.smithi019.stdout:Architecture: x86_64 2024-03-25T17:52:44.354 INFO:teuthology.orchestra.run.smithi019.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:52:44.354 INFO:teuthology.orchestra.run.smithi019.stdout:Group : Unspecified 2024-03-25T17:52:44.354 INFO:teuthology.orchestra.run.smithi019.stdout:Size : 8264254 2024-03-25T17:52:44.354 INFO:teuthology.orchestra.run.smithi019.stdout:License : GPLv3+ 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:Packager : builder@centos.org 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:Vendor : CentOS 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:Summary : Support tools for GRUB. 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:Description : 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout:hardware devices. 2024-03-25T17:52:44.355 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T17:52:44.356 INFO:teuthology.orchestra.run.smithi019.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:52:44.356 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:52:44.356 INFO:teuthology.task.kernel:Updating grub on smithi019 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:52:44.357 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:52:44.954 INFO:teuthology.orchestra.run.smithi019.stderr:Generating grub configuration file ... 2024-03-25T17:52:46.310 INFO:teuthology.orchestra.run.smithi019.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:52:46.350 INFO:teuthology.orchestra.run.smithi019.stderr:done 2024-03-25T17:52:46.352 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-03-25T17:52:46.367 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.GxAwZzEoBL 2024-03-25T17:52:46.367 DEBUG:teuthology.orchestra.run.smithi019:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GxAwZzEoBL 2024-03-25T17:52:46.458 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.GxAwZzEoBL 2024-03-25T17:52:46.560 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.GxAwZzEoBL is 6KB 2024-03-25T17:52:46.609 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.GxAwZzEoBL 2024-03-25T17:52:46.623 DEBUG:teuthology.orchestra.run.smithi019:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:52:46.691 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:52:46.692 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:52:46.692 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:52:46.692 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:52:46.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:52:46.842 DEBUG:teuthology.orchestra.run.smithi019:> sudo shutdown -r now 2024-03-25T17:53:13.918 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T17:53:13.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-03-25T17:53:13.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:53:16.846 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T17:53:16.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-25T17:53:16.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:53:48.575 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-03-25T17:53:57.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-03-25T17:53:57.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:15.967 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-03-25T17:54:16.850 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:54:25.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-25T17:54:25.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:26.208 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-03-25T17:54:26.778 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-25T17:54:26.778 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:54:26.778 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-03-25T17:54:26.793 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:54:26.793 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:54:26.793 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:54:26.793 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:54:27.794 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T17:54:27.795 DEBUG:teuthology.orchestra.run.smithi019:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:54:27.881 INFO:teuthology.orchestra.run.smithi019.stdout:ttyS1 2024-03-25T17:54:27.926 DEBUG:teuthology.parallel:result is None 2024-03-25T17:54:27.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-03-25T17:54:27.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:28.581 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-03-25T17:54:29.011 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-03-25T17:54:29.011 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:54:29.011 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-03-25T17:54:29.025 INFO:teuthology.orchestra.run.smithi188.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:54:29.025 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:54:29.025 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:54:29.025 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:54:30.026 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T17:54:30.026 DEBUG:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:54:30.082 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2024-03-25T17:54:30.123 DEBUG:teuthology.parallel:result is None 2024-03-25T17:54:30.123 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T17:54:30.131 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T17:54:30.131 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:54:30.134 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:54:30.148 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T17:54:30.156 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T17:54:30.166 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T17:54:30.166 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:54:30.194 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:54:30.223 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T17:54:30.231 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T17:54:30.232 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:54:30.255 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:54:30.294 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:54:30.296 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:54:30.307 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:54:30.309 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:54:30.311 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T17:54:30.317 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T17:54:30.318 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:54:30.356 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:54:30.382 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T17:54:30.393 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T17:54:30.393 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:54:30.427 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:54:30.441 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:54:30.526 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:54:30.574 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:54:30.659 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:54:30.704 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T17:54:30.704 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:54:30.776 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:54:30.817 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:54:30.868 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:54:30.949 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:54:30.991 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T17:54:30.991 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:54:31.063 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2024-03-25T17:54:31.065 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-03-25T17:54:31.102 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:54:31.153 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:54:31.291 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T17:54:31.301 INFO:teuthology.task.internal:Starting timer... 2024-03-25T17:54:31.301 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T17:54:31.319 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T17:54:31.332 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2024-03-25T17:54:31.388 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2024-03-25T17:54:31.390 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2024-03-25T17:54:31.437 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2024-03-25T17:54:31.439 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T17:54:31.439 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2024-03-25T17:54:31.461 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2024-03-25T17:54:31.461 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2024-03-25T17:54:31.484 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2024-03-25T17:54:31.485 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi019.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2024-03-25T17:54:31.485 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:54:31.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:54:31.532 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:54:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:54:31.559 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T17:54:31.559 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:54:31.613 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:54:31.652 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T17:54:31.664 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T17:54:31.681 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T17:54:31.685 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi019.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T17:58:59.533 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi019.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-03-25T17:58:59.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-25T17:58:59.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:58:59.628 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-03-25T17:58:59.717 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-25T17:58:59.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-03-25T17:58:59.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:58:59.806 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-03-25T17:58:59.889 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-03-25T17:58:59.889 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T17:58:59.900 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T17:58:59.900 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:58:59.901 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:58:59.903 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:58:59.903 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:58:59.934 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:58:59.951 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:58:59.959 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:58:59.975 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:58:59.980 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2024-03-25T17:58:59.994 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2024-03-25T17:59:00.005 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2024-03-25T17:59:00.010 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:59:00.018 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2024-03-25T17:59:00.027 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:59:00.035 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:59:00.051 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:59:00.078 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:59:00.081 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:59:00.081 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-03-25T17:59:00.081 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:00.081 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:00.081 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:00.081 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:00.108 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:59:00.111 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:59:00.111 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-03-25T17:59:00.111 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:00.111 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:00.111 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:00.111 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:00.112 INFO:teuthology.run_tasks:Running task install... 2024-03-25T17:59:00.120 DEBUG:teuthology.task.install:project ceph 2024-03-25T17:59:00.120 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T17:59:00.121 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T17:59:00.121 INFO:teuthology.task.install:Using flavor: default 2024-03-25T17:59:00.126 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T17:59:00.126 INFO:teuthology.task.install:extra packages: [] 2024-03-25T17:59:00.127 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:59:00.127 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:59:00.129 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:59:00.130 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:59:00.356 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:59:00.356 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:59:00.360 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:59:00.360 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:59:00.764 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:59:00.764 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T17:59:00.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:59:00.767 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:59:00.767 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T17:59:00.767 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:59:00.797 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:59:00.797 DEBUG:teuthology.orchestra.run.smithi019:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:59:00.798 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:59:00.798 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:59:00.867 DEBUG:teuthology.orchestra.run.smithi019:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:59:00.869 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:59:00.949 DEBUG:teuthology.orchestra.run.smithi019:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:59:00.951 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:59:01.017 INFO:teuthology.orchestra.run.smithi019.stdout:check_obsoletes = 1 2024-03-25T17:59:01.018 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2024-03-25T17:59:01.019 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum clean all 2024-03-25T17:59:01.019 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2024-03-25T17:59:01.267 INFO:teuthology.orchestra.run.smithi188.stdout:74 files removed 2024-03-25T17:59:01.279 INFO:teuthology.orchestra.run.smithi019.stdout:74 files removed 2024-03-25T17:59:01.294 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw 2024-03-25T17:59:01.306 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw 2024-03-25T17:59:02.699 INFO:teuthology.orchestra.run.smithi019.stdout:ceph packages for x86_64 75 kB/s | 82 kB 00:01 2024-03-25T17:59:02.759 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 70 kB/s | 82 kB 00:01 2024-03-25T17:59:03.474 INFO:teuthology.orchestra.run.smithi019.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-25T17:59:03.618 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T17:59:04.360 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-03-25T17:59:04.376 INFO:teuthology.orchestra.run.smithi019.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T17:59:05.146 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - AppStream 9.0 MB/s | 6.7 MB 00:00 2024-03-25T17:59:05.203 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 8.5 MB/s | 6.7 MB 00:00 2024-03-25T17:59:06.862 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-25T17:59:07.363 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - BaseOS 1.8 MB/s | 2.0 MB 00:01 2024-03-25T17:59:16.769 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - BaseOS 909 kB/s | 8.0 MB 00:09 2024-03-25T17:59:17.576 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 795 kB/s | 8.0 MB 00:10 2024-03-25T17:59:26.330 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - AppStream 2.2 MB/s | 19 MB 00:08 2024-03-25T17:59:26.940 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-03-25T17:59:38.125 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - CRB 762 kB/s | 6.1 MB 00:08 2024-03-25T17:59:39.593 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 701 kB/s | 6.1 MB 00:08 2024-03-25T17:59:40.578 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-25T17:59:41.139 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-25T17:59:41.974 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T17:59:43.260 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-25T17:59:47.179 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 4.7 kB/s | 1.7 kB 00:00 2024-03-25T17:59:48.324 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-25T17:59:49.649 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T17:59:49.661 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T17:59:49.662 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-03-25T17:59:49.662 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T17:59:49.662 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T17:59:49.662 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T17:59:49.662 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-03-25T17:59:49.662 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T17:59:49.663 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:59:49.663 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-03-25T17:59:49.663 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options 2024-03-25T17:59:49.663 INFO:teuthology.orchestra.run.smithi188.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T17:59:49.663 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T17:59:49.663 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:59:49.663 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T17:59:49.663 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T17:59:49.663 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T17:59:49.663 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T17:59:49.664 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout:Installing weak dependencies: 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T17:59:49.665 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T17:59:49.666 INFO:teuthology.orchestra.run.smithi188.stdout:Install 30 Packages 2024-03-25T17:59:49.666 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 2 Packages 2024-03-25T17:59:49.666 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T17:59:49.666 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 58 M 2024-03-25T17:59:49.666 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T17:59:50.866 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T17:59:50.883 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:59:50.884 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T17:59:50.885 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout:Installing weak dependencies: 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T17:59:50.886 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T17:59:50.887 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T17:59:50.887 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T17:59:50.887 INFO:teuthology.orchestra.run.smithi019.stdout:Install 30 Packages 2024-03-25T17:59:50.887 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 2 Packages 2024-03-25T17:59:50.887 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T17:59:50.887 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 58 M 2024-03-25T17:59:50.887 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:00:05.075 INFO:teuthology.orchestra.run.smithi019.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 395 kB/s | 5.3 MB 00:13 2024-03-25T18:00:05.177 INFO:teuthology.orchestra.run.smithi019.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T18:00:07.009 INFO:teuthology.orchestra.run.smithi019.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 405 kB/s | 742 kB 00:01 2024-03-25T18:00:08.167 INFO:teuthology.orchestra.run.smithi019.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 430 kB/s | 498 kB 00:01 2024-03-25T18:00:12.643 INFO:teuthology.orchestra.run.smithi188.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 243 kB/s | 5.3 MB 00:22 2024-03-25T18:00:13.626 INFO:teuthology.orchestra.run.smithi188.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 26 kB/s | 25 kB 00:00 2024-03-25T18:00:15.493 INFO:teuthology.orchestra.run.smithi188.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 397 kB/s | 742 kB 00:01 2024-03-25T18:00:17.276 INFO:teuthology.orchestra.run.smithi188.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 279 kB/s | 498 kB 00:01 2024-03-25T18:00:24.291 INFO:teuthology.orchestra.run.smithi019.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 324 kB/s | 5.1 MB 00:16 2024-03-25T18:00:24.542 INFO:teuthology.orchestra.run.smithi019.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 181 kB/s | 45 kB 00:00 2024-03-25T18:00:25.242 INFO:teuthology.orchestra.run.smithi019.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 188 kB/s | 131 kB 00:00 2024-03-25T18:00:26.133 INFO:teuthology.orchestra.run.smithi019.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 182 kB/s | 162 kB 00:00 2024-03-25T18:00:27.326 INFO:teuthology.orchestra.run.smithi019.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 269 kB/s | 320 kB 00:01 2024-03-25T18:00:28.450 INFO:teuthology.orchestra.run.smithi019.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 265 kB/s | 298 kB 00:01 2024-03-25T18:00:28.867 INFO:teuthology.orchestra.run.smithi019.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 238 kB/s | 99 kB 00:00 2024-03-25T18:00:28.976 INFO:teuthology.orchestra.run.smithi019.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-25T18:00:29.026 INFO:teuthology.orchestra.run.smithi019.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:00:29.060 INFO:teuthology.orchestra.run.smithi019.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:00:29.118 INFO:teuthology.orchestra.run.smithi019.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:00:29.160 INFO:teuthology.orchestra.run.smithi019.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:00:29.194 INFO:teuthology.orchestra.run.smithi019.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T18:00:29.359 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:29.472 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:29.514 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:29.604 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:29.679 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:29.784 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:30.300 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:30.366 INFO:teuthology.orchestra.run.smithi188.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 248 kB/s | 9.7 MB 00:40 2024-03-25T18:00:30.582 INFO:teuthology.orchestra.run.smithi188.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 209 kB/s | 45 kB 00:00 2024-03-25T18:00:30.626 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:30.691 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:30.710 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:30.796 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:30.962 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:31.045 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:31.206 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:31.242 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:00:31.262 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:31.324 INFO:teuthology.orchestra.run.smithi188.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 177 kB/s | 131 kB 00:00 2024-03-25T18:00:31.335 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:31.514 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:31.567 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:31.730 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:32.083 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:32.149 INFO:teuthology.orchestra.run.smithi188.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 196 kB/s | 162 kB 00:00 2024-03-25T18:00:32.251 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:32.626 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:32.712 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:00:32.856 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:00:33.002 INFO:teuthology.orchestra.run.smithi019.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 58 kB/s | 221 kB 00:03 2024-03-25T18:00:33.146 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:33.237 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:33.277 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:33.372 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:33.444 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:33.524 INFO:teuthology.orchestra.run.smithi188.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 233 kB/s | 320 kB 00:01 2024-03-25T18:00:33.559 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:33.669 INFO:teuthology.orchestra.run.smithi019.stdout:(19/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 235 kB/s | 9.7 MB 00:42 2024-03-25T18:00:33.853 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:33.934 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:33.957 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:33.977 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:34.000 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:34.023 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:34.097 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:34.450 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:34.506 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:34.523 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:34.589 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:34.613 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:34.691 INFO:teuthology.orchestra.run.smithi188.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 255 kB/s | 298 kB 00:01 2024-03-25T18:00:34.797 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:34.913 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:35.020 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:35.077 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:35.093 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:35.118 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:35.141 INFO:teuthology.orchestra.run.smithi188.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 221 kB/s | 99 kB 00:00 2024-03-25T18:00:35.151 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:35.152 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:00:35.169 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:35.208 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:35.226 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:35.268 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:35.328 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:35.345 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:00:35.383 INFO:teuthology.orchestra.run.smithi188.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 438 kB/s | 106 kB 00:00 2024-03-25T18:00:35.388 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:35.444 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:35.464 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:35.483 INFO:teuthology.orchestra.run.smithi188.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-03-25T18:00:35.486 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:35.525 INFO:teuthology.orchestra.run.smithi188.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T18:00:35.559 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:35.578 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:35.600 INFO:teuthology.orchestra.run.smithi188.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-25T18:00:35.642 INFO:teuthology.orchestra.run.smithi188.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:00:35.647 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:35.668 INFO:teuthology.orchestra.run.smithi188.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-25T18:00:35.736 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:35.752 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:35.867 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:36.084 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:36.102 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:36.114 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:36.262 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:36.276 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:36.448 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:36.489 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:36.578 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:36.617 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:36.642 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:36.692 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:00:36.717 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:00:36.818 INFO:teuthology.orchestra.run.smithi188.stdout:(18/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 267 kB/s | 5.1 MB 00:19 2024-03-25T18:00:36.832 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:00:36.858 INFO:teuthology.orchestra.run.smithi019.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:00:36.876 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:36.908 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:36.919 INFO:teuthology.orchestra.run.smithi019.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 15 kB/s | 47 kB 00:03 2024-03-25T18:00:36.936 INFO:teuthology.orchestra.run.smithi019.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 9.1 kB/s | 36 kB 00:03 2024-03-25T18:00:37.069 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:37.069 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:37.409 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:37.428 INFO:teuthology.orchestra.run.smithi019.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 607 kB/s | 309 kB 00:00 2024-03-25T18:00:37.440 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:37.460 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:37.481 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:37.573 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:37.686 INFO:teuthology.orchestra.run.smithi019.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 101 kB/s | 26 kB 00:00 2024-03-25T18:00:37.895 INFO:teuthology.orchestra.run.smithi019.stdout:(24/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-03-25T18:00:37.926 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:37.957 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:38.028 INFO:teuthology.orchestra.run.smithi019.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-03-25T18:00:38.070 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:38.147 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:38.192 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:38.237 INFO:teuthology.orchestra.run.smithi019.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 197 kB/s | 67 kB 00:00 2024-03-25T18:00:38.283 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:38.288 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:38.324 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:00:38.363 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:38.391 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:38.454 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:38.539 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:38.655 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:38.696 INFO:teuthology.orchestra.run.smithi019.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-25T18:00:38.746 INFO:teuthology.orchestra.run.smithi019.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 377 kB/s | 192 kB 00:00 2024-03-25T18:00:38.759 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:38.759 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:38.778 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:38.782 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:38.796 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:38.804 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:38.856 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:00:38.858 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:38.889 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:38.922 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:38.960 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:38.992 INFO:teuthology.orchestra.run.smithi188.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 66 kB/s | 221 kB 00:03 2024-03-25T18:00:38.993 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:39.010 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:39.063 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:39.076 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:39.101 INFO:teuthology.orchestra.run.smithi188.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 16 kB/s | 36 kB 00:02 2024-03-25T18:00:39.198 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:39.357 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:39.446 INFO:teuthology.orchestra.run.smithi019.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-25T18:00:39.685 INFO:teuthology.orchestra.run.smithi188.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 530 kB/s | 309 kB 00:00 2024-03-25T18:00:39.711 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:39.750 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:39.839 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:40.325 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:40.651 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:40.812 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:41.171 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:41.312 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:41.386 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:41.433 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:41.536 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:41.572 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:00:41.639 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:41.774 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:41.896 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:41.915 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:41.936 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:41.998 INFO:teuthology.orchestra.run.smithi188.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:42.093 INFO:teuthology.orchestra.run.smithi188.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 15 kB/s | 47 kB 00:03 2024-03-25T18:00:42.285 INFO:teuthology.orchestra.run.smithi188.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 136 kB/s | 26 kB 00:00 2024-03-25T18:00:42.468 INFO:teuthology.orchestra.run.smithi188.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 265 kB/s | 49 kB 00:00 2024-03-25T18:00:42.652 INFO:teuthology.orchestra.run.smithi188.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 368 kB/s | 67 kB 00:00 2024-03-25T18:00:43.319 INFO:teuthology.orchestra.run.smithi188.stdout:(26/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 383 kB/s | 20 MB 00:53 2024-03-25T18:00:43.777 INFO:teuthology.orchestra.run.smithi188.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 745 kB/s | 838 kB 00:01 2024-03-25T18:00:44.011 INFO:teuthology.orchestra.run.smithi188.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 277 kB/s | 192 kB 00:00 2024-03-25T18:00:45.303 INFO:teuthology.orchestra.run.smithi188.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 805 kB/s | 4.4 MB 00:05 2024-03-25T18:00:45.745 INFO:teuthology.orchestra.run.smithi188.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 823 kB/s | 1.6 MB 00:01 2024-03-25T18:00:48.488 INFO:teuthology.orchestra.run.smithi019.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 355 kB/s | 3.4 MB 00:09 2024-03-25T18:00:52.252 INFO:teuthology.orchestra.run.smithi188.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 419 kB/s | 3.4 MB 00:08 2024-03-25T18:00:52.796 INFO:teuthology.orchestra.run.smithi019.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 240 kB/s | 3.1 MB 00:13 2024-03-25T18:01:01.033 INFO:teuthology.orchestra.run.smithi188.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 204 kB/s | 3.1 MB 00:15 2024-03-25T18:01:01.041 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:01.042 INFO:teuthology.orchestra.run.smithi188.stdout:Total 830 kB/s | 58 MB 01:11 2024-03-25T18:01:01.075 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:01:01.116 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:01:01.116 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:01:01.598 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:01:01.598 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:01:03.099 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:01:03.169 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:01:03.205 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:01:03.240 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:01:03.278 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:01:03.294 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:01:03.436 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:01:03.499 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:01:03.654 INFO:teuthology.orchestra.run.smithi019.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 281 kB/s | 20 MB 01:12 2024-03-25T18:01:03.661 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:03.661 INFO:teuthology.orchestra.run.smithi019.stdout:Total 814 kB/s | 58 MB 01:12 2024-03-25T18:01:03.692 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:01:03.732 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:01:03.733 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:01:03.824 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:01:03.890 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:03.981 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:04.076 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:01:04.108 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:01:04.196 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:01:04.224 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:01:04.224 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:01:04.225 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:01:04.285 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:01:04.352 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:01:04.377 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:01:04.441 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:01:04.483 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:01:04.591 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:01:04.627 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:01:04.667 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:01:04.708 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:01:04.737 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:01:04.778 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:01:05.096 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:01:05.124 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:01:05.215 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:01:05.246 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:01:05.287 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:01:05.745 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:01:05.777 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:01:05.832 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:01:05.848 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:01:05.900 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:01:05.929 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:01:05.930 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:01:05.958 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:01:05.983 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:01:05.998 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:01:06.014 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:01:06.224 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:01:06.279 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:01:06.693 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:01:06.719 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:06.811 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:06.900 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:01:06.928 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:01:06.991 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:01:07.020 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:01:07.064 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:01:07.141 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:01:07.164 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:01:07.228 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:01:07.263 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:01:07.373 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:01:07.405 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:01:07.466 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:01:07.495 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:01:07.524 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:01:07.557 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:01:07.887 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:01:07.912 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:01:07.986 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:01:08.016 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:01:08.059 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:01:08.282 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:08.317 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:08.541 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:01:08.573 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:01:08.627 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:01:08.698 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:01:08.733 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:01:08.874 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:08.923 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:01:09.005 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:01:09.005 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:01:09.005 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:01:09.005 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:01:09.041 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:01:11.016 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:11.055 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:11.631 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:11.675 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:01:11.768 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:01:11.768 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:01:11.769 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:01:11.769 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:01:11.818 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:01:18.465 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:01:18.465 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2024-03-25T18:01:18.465 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2024-03-25T18:01:18.465 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2024-03-25T18:01:18.465 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2024-03-25T18:01:18.465 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2024-03-25T18:01:18.465 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /root 2024-03-25T18:01:18.465 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2024-03-25T18:01:18.465 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:01:19.386 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:01:19.409 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:01:19.409 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:01:19.409 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:01:19.409 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:01:19.409 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:01:19.409 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:01:19.411 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:01:19.530 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:01:19.530 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:01:20.647 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:01:20.648 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:01:20.648 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:01:20.648 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:01:20.648 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:01:20.648 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:01:20.648 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:01:20.648 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:20.648 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:01:20.648 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:01:20.648 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:01:20.648 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:01:20.648 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:01:20.649 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:01:20.650 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:01:20.650 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:01:20.650 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:01:20.650 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:01:20.650 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:01:20.650 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:01:21.488 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:01:21.488 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /sys 2024-03-25T18:01:21.488 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /proc 2024-03-25T18:01:21.488 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /mnt 2024-03-25T18:01:21.488 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /var/tmp 2024-03-25T18:01:21.488 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /home 2024-03-25T18:01:21.488 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /root 2024-03-25T18:01:21.488 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /tmp 2024-03-25T18:01:21.488 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:01:21.755 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:01:21.755 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:01:21.755 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-03-25T18:01:21.755 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.755 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:01:21.756 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.757 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:21.758 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:01:21.758 INFO:teuthology.orchestra.run.smithi188.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:01:21.758 INFO:teuthology.orchestra.run.smithi188.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:01:21.758 INFO:teuthology.orchestra.run.smithi188.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:01:21.758 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:01:21.758 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:01:21.758 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:01:22.119 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-test 2024-03-25T18:01:22.413 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:01:22.437 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:01:22.437 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:01:22.437 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:01:22.438 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:01:22.438 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:01:22.438 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:01:22.439 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:01:22.507 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:01:22.507 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:01:22.672 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:35 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:01:22.831 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:01:22.832 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:01:22.832 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-03-25T18:01:22.832 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:01:22.832 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:01:22.832 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:01:22.833 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-03-25T18:01:22.833 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:01:22.833 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:01:22.833 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:01:22.833 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:01:22.833 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:01:22.833 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:01:22.833 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:01:22.833 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:01:22.833 INFO:teuthology.orchestra.run.smithi188.stdout:Install 6 Packages 2024-03-25T18:01:22.833 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:01:22.834 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 48 M 2024-03-25T18:01:22.834 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 204 M 2024-03-25T18:01:22.834 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:01:23.143 INFO:teuthology.orchestra.run.smithi188.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T18:01:23.192 INFO:teuthology.orchestra.run.smithi188.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T18:01:23.217 INFO:teuthology.orchestra.run.smithi188.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-25T18:01:23.401 INFO:teuthology.orchestra.run.smithi188.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-25T18:01:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:01:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:01:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:01:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:01:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:01:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:01:23.713 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:01:23.714 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:01:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:01:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:01:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:01:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:01:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:01:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:01:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:01:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:01:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:01:23.715 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:01:23.716 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:01:23.716 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:01:23.716 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:01:23.716 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:01:24.251 INFO:teuthology.orchestra.run.smithi188.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 127 kB/s | 163 kB 00:01 2024-03-25T18:01:24.716 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:01:24.716 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:01:24.716 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2024-03-25T18:01:24.716 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.716 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:01:24.717 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:24.718 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:01:24.719 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:01:24.719 INFO:teuthology.orchestra.run.smithi019.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:01:24.719 INFO:teuthology.orchestra.run.smithi019.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:01:24.719 INFO:teuthology.orchestra.run.smithi019.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:01:24.719 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:01:24.719 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:01:24.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-test 2024-03-25T18:01:25.557 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:37 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:01:25.722 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:01:25.722 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:01:25.722 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:01:25.723 INFO:teuthology.orchestra.run.smithi019.stdout:Install 6 Packages 2024-03-25T18:01:25.724 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:01:25.724 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 48 M 2024-03-25T18:01:25.724 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 204 M 2024-03-25T18:01:25.724 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:01:25.936 INFO:teuthology.orchestra.run.smithi019.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-25T18:01:26.003 INFO:teuthology.orchestra.run.smithi019.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.9 MB/s | 300 kB 00:00 2024-03-25T18:01:26.036 INFO:teuthology.orchestra.run.smithi019.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T18:01:26.262 INFO:teuthology.orchestra.run.smithi019.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 832 kB/s | 187 kB 00:00 2024-03-25T18:01:27.020 INFO:teuthology.orchestra.run.smithi019.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 132 kB/s | 163 kB 00:01 2024-03-25T18:03:16.876 INFO:teuthology.orchestra.run.smithi019.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 436 kB/s | 47 MB 01:51 2024-03-25T18:03:16.878 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:16.878 INFO:teuthology.orchestra.run.smithi019.stdout:Total 444 kB/s | 48 MB 01:51 2024-03-25T18:03:16.889 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:03:16.910 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:03:16.910 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:03:17.300 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:03:17.300 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:03:17.899 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:03:17.976 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:03:18.030 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:03:18.070 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:03:18.115 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:03:18.145 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:03:22.859 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:03:22.938 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:03:25.286 INFO:teuthology.orchestra.run.smithi188.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 396 kB/s | 47 MB 02:02 2024-03-25T18:03:25.288 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:25.288 INFO:teuthology.orchestra.run.smithi188.stdout:Total 403 kB/s | 48 MB 02:02 2024-03-25T18:03:25.300 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:03:25.315 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:03:25.315 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:03:25.315 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:03:25.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:03:25.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:03:25.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:03:25.319 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:03:25.319 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:03:25.693 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:03:25.693 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:25.694 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:03:25.694 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:25.694 INFO:teuthology.orchestra.run.smithi019.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:03:25.694 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:25.694 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:03:25.694 INFO:teuthology.orchestra.run.smithi019.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:03:25.694 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:03:25.694 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:25.694 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:03:25.699 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:03:25.700 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:03:26.002 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph 2024-03-25T18:03:26.292 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:03:26.377 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:03:26.420 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:03:26.460 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:03:26.508 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:03:26.538 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:03:26.572 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:03:38 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:03:26.731 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:03:26.733 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================= 2024-03-25T18:03:26.733 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-25T18:03:26.733 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================= 2024-03-25T18:03:26.733 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:03:26.734 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:03:26.735 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:03:26.735 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:03:26.735 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:03:26.735 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:03:26.735 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:03:26.735 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:03:26.735 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:03:26.735 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:03:26.735 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:03:26.735 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:03:26.735 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:03:26.735 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:26.736 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:03:26.737 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================= 2024-03-25T18:03:26.737 INFO:teuthology.orchestra.run.smithi019.stdout:Install 36 Packages 2024-03-25T18:03:26.737 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:26.737 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 30 M 2024-03-25T18:03:26.737 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 107 M 2024-03-25T18:03:26.737 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:03:27.403 INFO:teuthology.orchestra.run.smithi019.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T18:03:30.212 INFO:teuthology.orchestra.run.smithi019.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 470 kB/s | 1.5 MB 00:03 2024-03-25T18:03:30.437 INFO:teuthology.orchestra.run.smithi019.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 627 kB/s | 2.1 MB 00:03 2024-03-25T18:03:30.537 INFO:teuthology.orchestra.run.smithi019.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.4 MB/s | 247 kB 00:00 2024-03-25T18:03:30.654 INFO:teuthology.orchestra.run.smithi019.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:03:30.696 INFO:teuthology.orchestra.run.smithi019.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:03:30.721 INFO:teuthology.orchestra.run.smithi019.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:03:30.746 INFO:teuthology.orchestra.run.smithi019.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T18:03:30.772 INFO:teuthology.orchestra.run.smithi019.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T18:03:30.814 INFO:teuthology.orchestra.run.smithi019.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T18:03:30.889 INFO:teuthology.orchestra.run.smithi019.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:03:30.923 INFO:teuthology.orchestra.run.smithi019.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:03:30.956 INFO:teuthology.orchestra.run.smithi019.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-25T18:03:30.998 INFO:teuthology.orchestra.run.smithi019.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:03:31.021 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:03:31.105 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:03:31.174 INFO:teuthology.orchestra.run.smithi019.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 714 kB/s | 125 kB 00:00 2024-03-25T18:03:31.340 INFO:teuthology.orchestra.run.smithi019.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 134 kB/s | 22 kB 00:00 2024-03-25T18:03:31.691 INFO:teuthology.orchestra.run.smithi019.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 432 kB/s | 151 kB 00:00 2024-03-25T18:03:31.766 INFO:teuthology.orchestra.run.smithi019.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 582 kB/s | 43 kB 00:00 2024-03-25T18:03:31.866 INFO:teuthology.orchestra.run.smithi019.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-25T18:03:32.025 INFO:teuthology.orchestra.run.smithi019.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-03-25T18:03:32.092 INFO:teuthology.orchestra.run.smithi019.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2024-03-25T18:03:32.292 INFO:teuthology.orchestra.run.smithi019.stdout:(22/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 8.1 MB/s | 17 MB 00:02 2024-03-25T18:03:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 78 kB/s | 18 kB 00:00 2024-03-25T18:03:32.385 INFO:teuthology.orchestra.run.smithi019.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 293 kB/s | 19 kB 00:00 2024-03-25T18:03:32.451 INFO:teuthology.orchestra.run.smithi019.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2024-03-25T18:03:32.527 INFO:teuthology.orchestra.run.smithi019.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 618 kB/s | 46 kB 00:00 2024-03-25T18:03:32.552 INFO:teuthology.orchestra.run.smithi019.stdout:(27/36): python3-jaraco-collections-3.0.0-8.el9 89 kB/s | 23 kB 00:00 2024-03-25T18:03:32.602 INFO:teuthology.orchestra.run.smithi019.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-25T18:03:32.669 INFO:teuthology.orchestra.run.smithi019.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 247 kB/s | 16 kB 00:00 2024-03-25T18:03:32.736 INFO:teuthology.orchestra.run.smithi019.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-25T18:03:32.778 INFO:teuthology.orchestra.run.smithi019.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-03-25T18:03:32.803 INFO:teuthology.orchestra.run.smithi019.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 535 kB/s | 36 kB 00:00 2024-03-25T18:03:32.870 INFO:teuthology.orchestra.run.smithi019.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-25T18:03:32.895 INFO:teuthology.orchestra.run.smithi019.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-25T18:03:32.929 INFO:teuthology.orchestra.run.smithi019.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-25T18:03:33.496 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:03:33.496 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:03:33.497 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:03:33.497 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:03:33.497 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:03:33.497 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:03:33.704 INFO:teuthology.orchestra.run.smithi019.stdout:(36/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 761 kB/s | 4.7 MB 00:06 2024-03-25T18:03:33.710 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:33.710 INFO:teuthology.orchestra.run.smithi019.stdout:Total 4.3 MB/s | 30 MB 00:06 2024-03-25T18:03:33.734 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:03:33.780 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:03:33.780 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:03:33.889 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:03:33.889 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:03:33.889 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:03:33.890 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:33.890 INFO:teuthology.orchestra.run.smithi188.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:03:33.890 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:33.890 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:03:33.890 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:03:33.890 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:03:33.890 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:03:33.890 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:03:34.205 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph 2024-03-25T18:03:34.209 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:03:34.210 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:03:34.777 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:03:47 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:03:34.942 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:03:34.944 INFO:teuthology.orchestra.run.smithi188.stdout:========================================================================================= 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout:========================================================================================= 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:03:34.945 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:03:34.946 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:03:34.947 INFO:teuthology.orchestra.run.smithi188.stdout:========================================================================================= 2024-03-25T18:03:34.948 INFO:teuthology.orchestra.run.smithi188.stdout:Install 36 Packages 2024-03-25T18:03:34.948 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:03:34.948 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 30 M 2024-03-25T18:03:34.948 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 107 M 2024-03-25T18:03:34.948 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:03:35.598 INFO:teuthology.orchestra.run.smithi188.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T18:03:35.676 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:03:35.732 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:03:35.774 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:03:35.801 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:03:35.874 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:03:35.916 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:03:35.949 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:03:36.045 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:03:36.122 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:03:36.169 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:03:36.242 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:03:36.295 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:03:36.333 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:03:36.391 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:03:36.454 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:03:36.648 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:03:36.703 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:03:36.742 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:03:36.777 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:03:36.806 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:03:36.886 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:03:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:03:36.950 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:03:36.990 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:03:37.026 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:03:37.059 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:03:37.153 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:03:37.199 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:03:37.274 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:03:37.415 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:03:37.549 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:03:37.715 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:03:37.737 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:03:37.761 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:03:37.761 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:03:37.761 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:03:37.762 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:03:37.762 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:03:37.762 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:39.288 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:03:39.314 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:03:39.315 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:03:39.315 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:03:39.315 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:03:39.315 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:03:39.315 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:39.448 INFO:teuthology.orchestra.run.smithi188.stdout:(2/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 4.7 MB 00:03 2024-03-25T18:03:39.780 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:03:39.803 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:03:39.803 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:03:39.803 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:03:39.803 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:03:39.803 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:03:39.803 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:40.022 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:03:40.045 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:03:40.045 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:03:40.045 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:03:40.045 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:03:40.046 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:03:40.046 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:40.114 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:03:41.098 INFO:teuthology.orchestra.run.smithi188.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 370 kB/s | 2.1 MB 00:05 2024-03-25T18:03:41.440 INFO:teuthology.orchestra.run.smithi188.stdout:(4/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 245 kB/s | 1.5 MB 00:06 2024-03-25T18:03:41.557 INFO:teuthology.orchestra.run.smithi188.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:03:41.607 INFO:teuthology.orchestra.run.smithi188.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-25T18:03:41.657 INFO:teuthology.orchestra.run.smithi188.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 633 kB/s | 32 kB 00:00 2024-03-25T18:03:41.699 INFO:teuthology.orchestra.run.smithi188.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-25T18:03:41.758 INFO:teuthology.orchestra.run.smithi188.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 664 kB/s | 39 kB 00:00 2024-03-25T18:03:41.783 INFO:teuthology.orchestra.run.smithi188.stdout:(10/36): ceph-mgr-modules-core-19.0.0-1439.g5dc 361 kB/s | 247 kB 00:00 2024-03-25T18:03:41.841 INFO:teuthology.orchestra.run.smithi188.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-25T18:03:41.921 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:03:41.921 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:03:41.921 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:03:41.921 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:03:41.921 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:03:41.921 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:03:41.922 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:03:41.922 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:03:41.922 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:03:41.922 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:03:41.922 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:03:41.922 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:03:41.922 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:03:41.922 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:03:41.922 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:03:41.922 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:03:41.923 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:03:41.923 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:03:41.923 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:03:41.923 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:03:41.923 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:03:41.923 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:03:41.923 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:03:41.923 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:03:41.923 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:03:41.923 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:03:41.923 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:03:41.924 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:03:41.924 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:03:41.924 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:03:41.924 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:03:41.924 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:03:41.924 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:03:41.924 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:03:41.924 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:03:41.924 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:03:41.925 INFO:teuthology.orchestra.run.smithi188.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 8.6 MB/s | 1.1 MB 00:00 2024-03-25T18:03:41.942 INFO:teuthology.orchestra.run.smithi188.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-03-25T18:03:41.967 INFO:teuthology.orchestra.run.smithi188.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T18:03:42.059 INFO:teuthology.orchestra.run.smithi188.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.6 MB/s | 187 kB 00:00 2024-03-25T18:03:42.643 INFO:teuthology.orchestra.run.smithi188.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-03-25T18:03:42.776 INFO:teuthology.orchestra.run.smithi188.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 154 kB/s | 125 kB 00:00 2024-03-25T18:03:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:03:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:42.989 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:03:42.989 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:03:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:03:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:03:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:03:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:03:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:03:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:03:42.990 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:03:42.991 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:03:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:03:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:03:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:03:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:03:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:03:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:03:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:03:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:03:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:03:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:03:42.992 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:42.992 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:03:43.168 INFO:teuthology.orchestra.run.smithi188.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 111 kB/s | 43 kB 00:00 2024-03-25T18:03:43.194 INFO:teuthology.orchestra.run.smithi188.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 274 kB/s | 151 kB 00:00 2024-03-25T18:03:43.320 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base 2024-03-25T18:03:43.377 INFO:teuthology.orchestra.run.smithi188.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 823 kB/s | 172 kB 00:00 2024-03-25T18:03:43.486 INFO:teuthology.orchestra.run.smithi188.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2024-03-25T18:03:43.603 INFO:teuthology.orchestra.run.smithi188.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 877 kB/s | 358 kB 00:00 2024-03-25T18:03:43.620 INFO:teuthology.orchestra.run.smithi188.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 132 kB/s | 18 kB 00:00 2024-03-25T18:03:43.711 INFO:teuthology.orchestra.run.smithi188.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-03-25T18:03:43.729 INFO:teuthology.orchestra.run.smithi188.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-25T18:03:43.829 INFO:teuthology.orchestra.run.smithi188.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 168 kB/s | 20 kB 00:00 2024-03-25T18:03:43.855 INFO:teuthology.orchestra.run.smithi188.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 370 kB/s | 46 kB 00:00 2024-03-25T18:03:43.891 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:03:55 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:03:43.963 INFO:teuthology.orchestra.run.smithi188.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 590 kB/s | 79 kB 00:00 2024-03-25T18:03:43.993 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:03:44.030 INFO:teuthology.orchestra.run.smithi188.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-03-25T18:03:44.047 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:03:44.049 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-25T18:03:44.049 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:03:44.105 INFO:teuthology.orchestra.run.smithi188.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 116 kB/s | 16 kB 00:00 2024-03-25T18:03:44.116 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install cephadm 2024-03-25T18:03:44.181 INFO:teuthology.orchestra.run.smithi188.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 599 kB/s | 90 kB 00:00 2024-03-25T18:03:44.239 INFO:teuthology.orchestra.run.smithi188.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 268 kB/s | 36 kB 00:00 2024-03-25T18:03:44.340 INFO:teuthology.orchestra.run.smithi188.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-03-25T18:03:44.448 INFO:teuthology.orchestra.run.smithi188.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.0 MB/s | 427 kB 00:00 2024-03-25T18:03:44.473 INFO:teuthology.orchestra.run.smithi188.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 150 kB/s | 20 kB 00:00 2024-03-25T18:03:44.682 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:03:56 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:03:44.838 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:03:44.839 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:03:44.839 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-25T18:03:44.839 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:03:44.839 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:03:44.839 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:03:44.839 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:44.839 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:03:44.839 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:03:44.840 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T18:03:44.840 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:44.840 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 747 k 2024-03-25T18:03:44.840 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 754 k 2024-03-25T18:03:44.840 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:03:47.921 INFO:teuthology.orchestra.run.smithi019.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 243 kB/s | 747 kB 00:03 2024-03-25T18:03:47.922 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:47.922 INFO:teuthology.orchestra.run.smithi019.stdout:Total 243 kB/s | 747 kB 00:03 2024-03-25T18:03:47.922 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:03:47.928 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:03:47.928 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:03:47.935 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:03:47.935 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:03:48.019 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:03:48.449 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:03:48.536 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:03:48.977 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:03:49.217 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:03:49.217 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:49.217 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:03:49.217 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:03:49.217 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:49.217 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:03:49.434 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:03:50.004 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:04:02 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:03:50.158 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:03:50.158 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:03:50.159 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T18:03:50.159 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:03:50.159 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:03:50.159 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:03:50.159 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:50.159 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:03:50.159 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:03:50.159 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T18:03:50.159 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:50.160 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 145 k 2024-03-25T18:03:50.160 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 436 k 2024-03-25T18:03:50.160 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:03:51.076 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 159 kB/s | 145 kB 00:00 2024-03-25T18:03:51.077 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:51.077 INFO:teuthology.orchestra.run.smithi019.stdout:Total 158 kB/s | 145 kB 00:00 2024-03-25T18:03:51.078 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:03:51.087 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:03:51.088 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:03:51.146 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:03:51.147 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:03:51.293 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:03:51.385 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:03:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:03:51.410 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:03:51.410 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:03:51.410 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:52.383 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:03:52.384 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:52.384 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:03:52.384 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:52.384 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:52.384 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:03:52.607 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr 2024-03-25T18:03:53.186 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:04:05 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:03:53.288 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:03:53.344 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:03:53.345 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-25T18:03:53.345 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:03:53.413 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:03:53.997 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:04:05 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:03:54.153 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:03:54.154 INFO:teuthology.orchestra.run.smithi019.stdout:====================================================================================== 2024-03-25T18:03:54.154 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-25T18:03:54.154 INFO:teuthology.orchestra.run.smithi019.stdout:====================================================================================== 2024-03-25T18:03:54.154 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:03:54.154 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:03:54.155 INFO:teuthology.orchestra.run.smithi019.stdout:====================================================================================== 2024-03-25T18:03:54.156 INFO:teuthology.orchestra.run.smithi019.stdout:Install 12 Packages 2024-03-25T18:03:54.156 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:03:54.156 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 6.9 M 2024-03-25T18:03:54.156 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 26 M 2024-03-25T18:03:54.156 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:03:54.705 INFO:teuthology.orchestra.run.smithi019.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 71 kB/s | 31 kB 00:00 2024-03-25T18:03:54.747 INFO:teuthology.orchestra.run.smithi019.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 32 kB/s | 15 kB 00:00 2024-03-25T18:03:54.897 INFO:teuthology.orchestra.run.smithi019.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-25T18:03:54.931 INFO:teuthology.orchestra.run.smithi019.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-25T18:03:55.606 INFO:teuthology.orchestra.run.smithi019.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.2 MB/s | 863 kB 00:00 2024-03-25T18:03:55.706 INFO:teuthology.orchestra.run.smithi019.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 707 kB/s | 548 kB 00:00 2024-03-25T18:03:55.882 INFO:teuthology.orchestra.run.smithi019.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-03-25T18:03:56.124 INFO:teuthology.orchestra.run.smithi019.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 598 kB/s | 144 kB 00:00 2024-03-25T18:03:56.224 INFO:teuthology.orchestra.run.smithi019.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 307 kB/s | 31 kB 00:00 2024-03-25T18:03:56.475 INFO:teuthology.orchestra.run.smithi019.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 754 kB/s | 188 kB 00:00 2024-03-25T18:03:57.541 INFO:teuthology.orchestra.run.smithi019.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 1.1 MB/s | 2.0 MB 00:01 2024-03-25T18:03:59.033 INFO:teuthology.orchestra.run.smithi019.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 382 kB/s | 1.8 MB 00:04 2024-03-25T18:03:59.037 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:59.038 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.4 MB/s | 6.9 MB 00:04 2024-03-25T18:03:59.058 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:03:59.077 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:03:59.078 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:03:59.270 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:03:59.270 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:03:59.921 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:04:00.001 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:04:00.044 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:04:00.092 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:04:00.129 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:04:00.235 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:04:00.369 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:04:00.460 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:04:00.518 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:04:00.546 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:04:00.586 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:04:00.850 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:04:00.877 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:04:01.790 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:04:01.790 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:04:01.790 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:04:01.790 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:04:01.790 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:04:01.790 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:04:01.791 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:04:01.791 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:04:01.791 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:04:01.791 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:04:01.791 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:04:01.791 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:04:02.216 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:04:02.217 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:02.217 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:04:02.217 INFO:teuthology.orchestra.run.smithi019.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:04:02.217 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:02.217 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:02.217 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:02.217 INFO:teuthology.orchestra.run.smithi019.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:04:02.217 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:04:02.217 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:04:02.217 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:04:02.218 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:04:02.218 INFO:teuthology.orchestra.run.smithi019.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:04:02.218 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:04:02.218 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:04:02.218 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:02.218 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:04:02.455 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:04:03.023 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:04:15 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:04:03.179 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:04:03.180 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2024-03-25T18:04:03.181 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-25T18:04:03.181 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2024-03-25T18:04:03.181 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:04:03.181 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:04:03.181 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-25T18:04:03.181 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:04:03.181 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:04:03.181 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:04:03.181 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:04:03.181 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:04:03.181 INFO:teuthology.orchestra.run.smithi019.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:04:03.181 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:04:03.182 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:04:03.182 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:04:03.182 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:04:03.182 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:04:03.182 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:03.182 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:04:03.182 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2024-03-25T18:04:03.182 INFO:teuthology.orchestra.run.smithi019.stdout:Install 12 Packages 2024-03-25T18:04:03.182 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:03.183 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 37 M 2024-03-25T18:04:03.183 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 209 M 2024-03-25T18:04:03.183 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:04:03.456 INFO:teuthology.orchestra.run.smithi019.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 268 kB/s | 32 kB 00:00 2024-03-25T18:04:03.514 INFO:teuthology.orchestra.run.smithi019.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 502 kB/s | 17 kB 00:00 2024-03-25T18:04:03.548 INFO:teuthology.orchestra.run.smithi019.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:04:03.698 INFO:teuthology.orchestra.run.smithi019.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 32 MB/s | 4.7 MB 00:00 2024-03-25T18:04:03.832 INFO:teuthology.orchestra.run.smithi019.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T18:04:03.899 INFO:teuthology.orchestra.run.smithi019.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.3 MB/s | 3.0 MB 00:00 2024-03-25T18:04:03.924 INFO:teuthology.orchestra.run.smithi019.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T18:04:04.233 INFO:teuthology.orchestra.run.smithi019.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-25T18:04:05.108 INFO:teuthology.orchestra.run.smithi019.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 223 kB/s | 195 kB 00:00 2024-03-25T18:04:05.209 INFO:teuthology.orchestra.run.smithi019.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-25T18:04:05.675 INFO:teuthology.orchestra.run.smithi019.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 459 kB/s | 803 kB 00:01 2024-03-25T18:04:19.733 INFO:teuthology.orchestra.run.smithi188.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 426 kB/s | 17 MB 00:40 2024-03-25T18:04:19.739 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:19.739 INFO:teuthology.orchestra.run.smithi188.stdout:Total 678 kB/s | 30 MB 00:44 2024-03-25T18:04:19.763 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:04:19.804 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:04:19.804 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:04:20.244 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:04:20.244 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:04:21.613 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:04:21.685 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:04:21.724 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:04:21.754 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:04:21.825 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:04:21.870 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:04:21.903 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:04:21.994 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:04:22.072 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:04:22.122 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:04:22.194 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:04:22.248 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:04:22.286 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:04:22.335 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:04:22.390 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:04:22.574 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:04:22.624 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:04:22.680 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:04:22.714 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:04:22.744 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:04:22.821 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:04:22.855 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:04:22.888 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:04:22.924 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:04:22.959 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:04:22.993 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:04:23.091 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:04:23.133 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:04:23.205 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:04:23.342 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:04:23.459 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:04:23.624 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:04:23.655 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:04:23.680 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:04:23.680 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:23.680 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:04:23.680 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:04:23.681 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:04:23.681 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:25.157 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:04:25.185 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:04:25.185 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:25.186 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:04:25.186 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:04:25.186 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:04:25.186 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:25.649 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:04:25.673 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:04:25.673 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:25.673 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:04:25.673 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:04:25.673 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:04:25.674 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:25.883 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:04:25.906 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:04:25.906 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:25.907 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:04:25.907 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:04:25.907 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:04:25.907 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:25.972 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:04:27.663 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:04:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:04:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:04:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:04:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:04:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:04:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:04:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:04:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:04:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:04:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:04:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:04:27.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:04:27.665 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:04:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:04:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:04:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:04:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:04:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:04:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:04:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:04:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:04:27.666 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:04:27.798 INFO:teuthology.orchestra.run.smithi019.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 309 kB/s | 7.4 MB 00:24 2024-03-25T18:04:27.802 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:27.802 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.5 MB/s | 37 MB 00:24 2024-03-25T18:04:27.844 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:04:27.870 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:04:27.871 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:04:28.251 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:04:28.251 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:04:28.585 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:04:28.585 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:28.585 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:04:28.585 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:28.585 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:28.585 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:28.586 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:28.586 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:28.586 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:28.586 INFO:teuthology.orchestra.run.smithi188.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:04:28.586 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:04:28.586 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:04:28.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:04:28.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:04:28.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:04:28.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:04:28.586 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:04:28.587 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:04:28.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:04:28.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:04:28.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:04:28.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:04:28.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:04:28.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:04:28.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:04:28.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:04:28.588 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:04:28.588 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:28.588 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:04:28.942 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-base 2024-03-25T18:04:28.976 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:04:29.046 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:04:29.077 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:04:29.182 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:04:29.222 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:04:29.406 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:04:29.430 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:04:29.510 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:04:42 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:04:29.519 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:04:29.609 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:04:29.661 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:04:29.662 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-03-25T18:04:29.662 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:04:29.725 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install cephadm 2024-03-25T18:04:30.210 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:04:30.287 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:04:30.330 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:04:30.437 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:04:30.438 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:04:30.438 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-03-25T18:04:30.439 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:04:30.439 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:04:30.439 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:04:30.439 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:30.439 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:04:30.439 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:04:30.439 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T18:04:30.439 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:30.439 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 747 k 2024-03-25T18:04:30.439 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 754 k 2024-03-25T18:04:30.439 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:04:31.639 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:04:32.713 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:04:32.737 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:04:34.240 INFO:teuthology.orchestra.run.smithi188.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 197 kB/s | 747 kB 00:03 2024-03-25T18:04:34.240 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:34.241 INFO:teuthology.orchestra.run.smithi188.stdout:Total 197 kB/s | 747 kB 00:03 2024-03-25T18:04:34.241 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:04:34.247 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:04:34.247 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:04:34.253 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:04:34.253 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:04:34.329 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:04:34.692 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:04:34.780 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:04:35.144 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:04:35.315 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:04:35.315 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:04:35.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:04:35.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:04:35.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:04:35.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:04:35.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:04:35.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:04:35.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:04:35.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:04:35.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:04:35.316 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:04:35.451 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:04:35.451 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:35.451 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:04:35.451 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:35.451 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:35.451 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:04:35.683 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:04:35.798 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:04:35.798 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:35.798 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:04:35.798 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:35.798 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:04:35.798 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:04:35.798 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:04:35.799 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:04:35.799 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:04:35.799 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:04:35.799 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:04:35.799 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:04:35.799 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:04:35.799 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:04:35.799 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:04:35.799 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:35.799 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:04:36.184 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:04:36.261 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:04:49 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:04:36.410 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:04:36.411 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:04:36.411 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T18:04:36.411 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:04:36.411 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:04:36.411 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:04:36.411 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:36.411 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:04:36.411 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:04:36.411 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T18:04:36.411 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:36.412 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 145 k 2024-03-25T18:04:36.412 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 436 k 2024-03-25T18:04:36.412 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:04:36.758 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:04:36.915 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:04:36.916 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-25T18:04:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-25T18:04:36.916 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-25T18:04:36.916 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:04:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:04:36.916 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-25T18:04:36.916 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-25T18:04:36.917 INFO:teuthology.orchestra.run.smithi019.stdout:Install 13 Packages 2024-03-25T18:04:36.918 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:36.918 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.0 M 2024-03-25T18:04:36.918 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 26 M 2024-03-25T18:04:36.918 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:04:37.090 INFO:teuthology.orchestra.run.smithi019.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 146 kB/s | 18 kB 00:00 2024-03-25T18:04:37.140 INFO:teuthology.orchestra.run.smithi019.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 140 kB/s | 24 kB 00:00 2024-03-25T18:04:37.190 INFO:teuthology.orchestra.run.smithi019.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T18:04:37.224 INFO:teuthology.orchestra.run.smithi019.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-25T18:04:37.249 INFO:teuthology.orchestra.run.smithi019.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T18:04:37.274 INFO:teuthology.orchestra.run.smithi019.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 870 kB/s | 43 kB 00:00 2024-03-25T18:04:37.321 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 160 kB/s | 145 kB 00:00 2024-03-25T18:04:37.322 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:37.322 INFO:teuthology.orchestra.run.smithi188.stdout:Total 160 kB/s | 145 kB 00:00 2024-03-25T18:04:37.323 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:04:37.330 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:04:37.330 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:04:37.350 INFO:teuthology.orchestra.run.smithi019.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 200 kB/s | 15 kB 00:00 2024-03-25T18:04:37.375 INFO:teuthology.orchestra.run.smithi019.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 257 kB/s | 32 kB 00:00 2024-03-25T18:04:37.385 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:04:37.386 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:04:37.467 INFO:teuthology.orchestra.run.smithi019.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.7 MB/s | 203 kB 00:00 2024-03-25T18:04:37.509 INFO:teuthology.orchestra.run.smithi019.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-03-25T18:04:37.523 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:04:37.559 INFO:teuthology.orchestra.run.smithi019.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-03-25T18:04:37.589 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:04:37.613 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:04:37.613 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:37.613 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:04:37.613 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi019.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2024-03-25T18:04:37.718 INFO:teuthology.orchestra.run.smithi019.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 69 kB/s | 52 kB 00:00 2024-03-25T18:04:37.720 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:37.720 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-03-25T18:04:37.720 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:04:37.735 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:04:37.735 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:04:37.831 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:04:37.833 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:04:38.361 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:04:38.539 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:04:38.576 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:04:38.576 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:38.576 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:04:38.576 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:38.576 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:38.576 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:04:38.603 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:04:38.661 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:04:38.695 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:04:38.737 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:04:38.835 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:04:38.880 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr 2024-03-25T18:04:38.943 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:04:38.998 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:04:39.458 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:04:39.557 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:04:39.609 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:04:39.611 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-03-25T18:04:39.611 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:04:39.673 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:04:39.761 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:04:39.802 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:04:39.836 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:04:39.885 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:04:39.914 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:04:40.229 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:04:53 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:04:40.381 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:04:40.382 INFO:teuthology.orchestra.run.smithi188.stdout:====================================================================================== 2024-03-25T18:04:40.382 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout:====================================================================================== 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:04:40.383 INFO:teuthology.orchestra.run.smithi188.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:04:40.384 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:04:40.384 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:04:40.384 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:40.384 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:04:40.384 INFO:teuthology.orchestra.run.smithi188.stdout:====================================================================================== 2024-03-25T18:04:40.384 INFO:teuthology.orchestra.run.smithi188.stdout:Install 12 Packages 2024-03-25T18:04:40.384 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:40.385 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 6.9 M 2024-03-25T18:04:40.385 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 26 M 2024-03-25T18:04:40.385 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:04:40.728 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:04:40.728 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:04:40.729 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:04:40.729 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:04:40.729 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:04:40.729 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:04:40.729 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:04:40.729 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:04:40.729 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:04:40.729 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:04:40.729 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:04:40.729 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:04:40.729 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:04:41.129 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:04:41.129 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:41.129 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:04:41.129 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:41.129 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:04:41.129 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:04:41.129 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:04:41.129 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:04:41.129 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:04:41.130 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:04:41.130 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:04:41.130 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:04:41.130 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:04:41.130 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:04:41.130 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:04:41.130 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:04:41.130 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:41.130 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:04:41.276 INFO:teuthology.orchestra.run.smithi188.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 33 kB/s | 15 kB 00:00 2024-03-25T18:04:41.357 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:04:41.443 INFO:teuthology.orchestra.run.smithi188.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 50 kB/s | 31 kB 00:00 2024-03-25T18:04:41.510 INFO:teuthology.orchestra.run.smithi188.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-25T18:04:41.535 INFO:teuthology.orchestra.run.smithi188.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-25T18:04:41.827 INFO:teuthology.orchestra.run.smithi188.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T18:04:41.930 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:04:53 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:04:41.944 INFO:teuthology.orchestra.run.smithi188.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-03-25T18:04:42.085 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:04:42.086 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-25T18:04:42.086 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-25T18:04:42.087 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-25T18:04:42.087 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:04:42.087 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:04:42.087 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-25T18:04:42.087 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:04:42.087 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:04:42.087 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:04:42.087 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:04:42.087 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:04:42.087 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:42.087 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:04:42.087 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-25T18:04:42.088 INFO:teuthology.orchestra.run.smithi019.stdout:Install 6 Packages 2024-03-25T18:04:42.088 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:42.088 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 6.8 M 2024-03-25T18:04:42.088 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 33 M 2024-03-25T18:04:42.088 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:04:42.186 INFO:teuthology.orchestra.run.smithi188.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 8.4 MB/s | 2.0 MB 00:00 2024-03-25T18:04:42.219 INFO:teuthology.orchestra.run.smithi188.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.2 MB/s | 863 kB 00:00 2024-03-25T18:04:42.270 INFO:teuthology.orchestra.run.smithi188.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-03-25T18:04:42.412 INFO:teuthology.orchestra.run.smithi188.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-03-25T18:04:42.437 INFO:teuthology.orchestra.run.smithi188.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 141 kB/s | 31 kB 00:00 2024-03-25T18:04:42.520 INFO:teuthology.orchestra.run.smithi019.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.4 MB/s | 248 kB 00:00 2024-03-25T18:04:43.145 INFO:teuthology.orchestra.run.smithi019.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.9 MB/s | 5.8 MB 00:00 2024-03-25T18:04:43.462 INFO:teuthology.orchestra.run.smithi019.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 132 kB/s | 138 kB 00:01 2024-03-25T18:04:43.696 INFO:teuthology.orchestra.run.smithi019.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 105 kB/s | 58 kB 00:00 2024-03-25T18:04:43.737 INFO:teuthology.orchestra.run.smithi019.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 459 kB/s | 548 kB 00:01 2024-03-25T18:04:44.004 INFO:teuthology.orchestra.run.smithi019.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 94 kB/s | 51 kB 00:00 2024-03-25T18:04:44.009 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:44.009 INFO:teuthology.orchestra.run.smithi019.stdout:Total 3.5 MB/s | 6.8 MB 00:01 2024-03-25T18:04:44.024 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:04:44.035 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:04:44.036 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:04:44.129 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:04:44.130 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:04:44.421 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:04:44.594 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:04:44.640 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:04:45.193 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:04:45.278 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:04:45.347 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:04:45.376 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:04:46.170 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:04:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:04:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:04:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:04:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:04:46.171 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:04:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:04:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:46.478 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:04:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:04:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:04:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:04:46.478 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:04:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:04:46.479 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:46.479 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:04:46.685 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-fuse 2024-03-25T18:04:47.253 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:04:47.408 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:04:47.409 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:04:47.409 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-25T18:04:47.409 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:04:47.409 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:04:47.409 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:04:47.409 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-25T18:04:47.410 INFO:teuthology.orchestra.run.smithi019.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:04:47.410 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:47.410 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:04:47.410 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:04:47.410 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-03-25T18:04:47.410 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:47.410 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 964 k 2024-03-25T18:04:47.410 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 2.8 M 2024-03-25T18:04:47.410 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:04:47.523 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 699 kB/s | 79 kB 00:00 2024-03-25T18:04:49.052 INFO:teuthology.orchestra.run.smithi188.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 221 kB/s | 1.8 MB 00:08 2024-03-25T18:04:49.056 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:49.056 INFO:teuthology.orchestra.run.smithi188.stdout:Total 809 kB/s | 6.9 MB 00:08 2024-03-25T18:04:49.076 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:04:49.095 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:04:49.096 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:04:49.275 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:04:49.276 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:04:49.789 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:04:49.877 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:04:49.921 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:04:49.967 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:04:49.998 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:04:50.100 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:04:50.231 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:04:50.325 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:04:50.378 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:04:50.407 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:04:50.438 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:04:50.679 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:04:50.714 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:04:51.532 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:04:51.532 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:04:51.532 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:04:51.532 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:04:51.532 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:04:51.532 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:04:51.532 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:04:51.532 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:04:51.532 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:04:51.532 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:04:51.532 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:04:51.532 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:04:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:04:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:51.881 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:04:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:04:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:04:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:04:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:04:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:04:51.881 INFO:teuthology.orchestra.run.smithi188.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:04:51.882 INFO:teuthology.orchestra.run.smithi188.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:04:51.882 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:04:51.882 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:04:51.882 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:51.882 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:04:51.956 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 195 kB/s | 885 kB 00:04 2024-03-25T18:04:51.957 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:51.958 INFO:teuthology.orchestra.run.smithi019.stdout:Total 212 kB/s | 964 kB 00:04 2024-03-25T18:04:51.958 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:04:51.983 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:04:51.983 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:04:52.063 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:04:52.064 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:04:52.131 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:04:52.251 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:04:52.403 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:04:52.483 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:04:52.686 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:05:05 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:04:52.837 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:04:52.838 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-03-25T18:04:52.838 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-03-25T18:04:52.838 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-03-25T18:04:52.838 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:04:52.838 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:04:52.838 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-03-25T18:04:52.838 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:04:52.838 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:04:52.838 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout:Install 12 Packages 2024-03-25T18:04:52.839 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:04:52.840 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 37 M 2024-03-25T18:04:52.840 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 209 M 2024-03-25T18:04:52.840 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:04:53.102 INFO:teuthology.orchestra.run.smithi188.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 222 kB/s | 32 kB 00:00 2024-03-25T18:04:53.143 INFO:teuthology.orchestra.run.smithi188.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 414 kB/s | 17 kB 00:00 2024-03-25T18:04:53.185 INFO:teuthology.orchestra.run.smithi188.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 834 kB/s | 34 kB 00:00 2024-03-25T18:04:53.203 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:04:53.203 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:04:53.369 INFO:teuthology.orchestra.run.smithi188.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-25T18:04:53.519 INFO:teuthology.orchestra.run.smithi188.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-25T18:04:53.542 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:04:53.542 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:53.542 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:04:53.543 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:04:53.543 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:53.543 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:04:53.561 INFO:teuthology.orchestra.run.smithi188.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-25T18:04:53.695 INFO:teuthology.orchestra.run.smithi188.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-03-25T18:04:53.803 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-volume 2024-03-25T18:04:53.845 INFO:teuthology.orchestra.run.smithi188.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.3 MB/s | 803 kB 00:00 2024-03-25T18:04:53.996 INFO:teuthology.orchestra.run.smithi188.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-25T18:04:54.104 INFO:teuthology.orchestra.run.smithi188.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 758 kB/s | 195 kB 00:00 2024-03-25T18:04:54.379 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:06 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:04:54.529 INFO:teuthology.orchestra.run.smithi188.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 460 kB/s | 245 kB 00:00 2024-03-25T18:04:54.535 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:04:54.535 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:04:54.535 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-25T18:04:54.536 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:04:54.536 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:04:54.536 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:04:54.536 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-25T18:04:54.536 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:04:54.536 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:54.536 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:04:54.536 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:04:54.536 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-03-25T18:04:54.536 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:54.537 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 332 k 2024-03-25T18:04:54.537 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.5 M 2024-03-25T18:04:54.537 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:04:54.652 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 599 kB/s | 69 kB 00:00 2024-03-25T18:04:56.036 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 176 kB/s | 264 kB 00:01 2024-03-25T18:04:56.037 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:56.038 INFO:teuthology.orchestra.run.smithi019.stdout:Total 221 kB/s | 332 kB 00:01 2024-03-25T18:04:56.038 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:04:56.048 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:04:56.048 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:04:56.095 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:04:56.095 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:04:56.254 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:04:56.436 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:04:56.488 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:04:56.503 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:04:56.503 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:56.503 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:04:56.503 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:57.238 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:04:57.567 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:04:57.567 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:57.567 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:04:57.567 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:57.567 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:04:57.567 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:57.568 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:04:57.876 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados-devel 2024-03-25T18:04:58.454 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:10 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:04:58.610 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:04:58.610 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:04:58.611 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T18:04:58.611 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:04:58.611 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:04:58.611 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:04:58.611 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:58.611 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:04:58.611 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:04:58.611 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T18:04:58.611 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:04:58.612 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 127 k 2024-03-25T18:04:58.612 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 456 k 2024-03-25T18:04:58.612 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:05:00.208 INFO:teuthology.orchestra.run.smithi019.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 79 kB/s | 127 kB 00:01 2024-03-25T18:05:00.208 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:00.209 INFO:teuthology.orchestra.run.smithi019.stdout:Total 79 kB/s | 127 kB 00:01 2024-03-25T18:05:00.209 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:05:00.216 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:05:00.216 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:05:00.279 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:05:00.279 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:05:00.426 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:05:00.526 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:00.963 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:01.249 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:01.249 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:01.249 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:05:01.249 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:01.249 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:01.249 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:05:01.468 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs2 2024-03-25T18:05:02.042 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:14 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:05:02.143 INFO:teuthology.orchestra.run.smithi019.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:02.199 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:05:02.201 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-25T18:05:02.201 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:05:02.263 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs-devel 2024-03-25T18:05:02.838 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:14 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:05:02.992 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:05:02.993 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T18:05:02.993 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:02.993 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:05:02.993 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:05:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:02.994 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:05:02.994 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:02.994 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T18:05:02.994 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:02.994 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 31 k 2024-03-25T18:05:02.994 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 140 k 2024-03-25T18:05:02.994 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:05:03.527 INFO:teuthology.orchestra.run.smithi019.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 58 kB/s | 31 kB 00:00 2024-03-25T18:05:03.528 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:03.528 INFO:teuthology.orchestra.run.smithi019.stdout:Total 58 kB/s | 31 kB 00:00 2024-03-25T18:05:03.528 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:05:03.532 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:05:03.532 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:05:03.571 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:05:03.572 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:05:03.701 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:05:03.804 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:04.206 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:04.483 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:04.483 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:04.484 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:05:04.484 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:04.484 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:04.484 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:05:04.656 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados2 2024-03-25T18:05:05.233 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:17 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:05:05.333 INFO:teuthology.orchestra.run.smithi019.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:05.389 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:05:05.391 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-25T18:05:05.391 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:05:05.447 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librbd1 2024-03-25T18:05:06.022 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:18 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:05:06.124 INFO:teuthology.orchestra.run.smithi019.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:06.180 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:05:06.181 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-25T18:05:06.181 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:05:06.232 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rados 2024-03-25T18:05:06.806 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:18 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:05:06.906 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:06.960 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:05:06.961 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-25T18:05:06.961 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:05:07.022 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rgw 2024-03-25T18:05:07.594 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:05:07.694 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:07.749 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:05:07.750 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-25T18:05:07.750 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:05:07.801 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-cephfs 2024-03-25T18:05:08.374 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:05:08.473 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:08.528 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:05:08.529 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-25T18:05:08.529 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:05:08.576 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rbd 2024-03-25T18:05:09.148 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:21 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:05:09.248 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:09.302 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:05:09.303 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-25T18:05:09.303 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:05:09.352 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-fuse 2024-03-25T18:05:09.927 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:21 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:05:10.082 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:05:10.083 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:10.083 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-03-25T18:05:10.083 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:10.083 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:05:10.083 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:05:10.084 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:10.084 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:05:10.084 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:10.084 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T18:05:10.084 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:10.084 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 85 k 2024-03-25T18:05:10.084 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 231 k 2024-03-25T18:05:10.084 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:05:10.351 INFO:teuthology.orchestra.run.smithi188.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 435 kB/s | 7.4 MB 00:17 2024-03-25T18:05:10.354 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:10.355 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.1 MB/s | 37 MB 00:17 2024-03-25T18:05:10.396 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:05:10.421 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:05:10.421 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:05:10.782 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:05:10.782 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:05:10.822 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 115 kB/s | 85 kB 00:00 2024-03-25T18:05:10.823 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:10.823 INFO:teuthology.orchestra.run.smithi019.stdout:Total 115 kB/s | 85 kB 00:00 2024-03-25T18:05:10.823 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:05:10.831 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:05:10.831 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:05:10.891 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:05:10.891 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:05:11.032 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:05:11.132 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:11.503 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:11.514 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:05:11.590 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:05:11.630 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:05:11.723 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:05:11.767 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:05:11.799 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:11.799 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:11.799 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:05:11.800 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:11.800 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:11.800 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:05:11.943 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:05:11.966 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:05:12.019 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-mirror 2024-03-25T18:05:12.054 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:05:12.597 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:05:12.697 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:05:12.751 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:05:12.752 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:12.752 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T18:05:12.752 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:12.752 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:05:12.752 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:05:12.752 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:12.752 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:05:12.752 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:12.752 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T18:05:12.752 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:12.753 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 3.1 M 2024-03-25T18:05:12.753 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 13 M 2024-03-25T18:05:12.753 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:05:12.819 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:05:14.051 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:05:15.085 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:05:15.166 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:05:17.621 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:05:17.621 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:05:17.621 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:05:17.621 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:05:17.621 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:05:17.621 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:05:17.621 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:05:17.621 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:05:17.621 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:05:17.621 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:05:17.621 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:05:17.621 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:05:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:05:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:18.099 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:05:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:05:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:05:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:05:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:05:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:05:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:05:18.099 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:05:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:05:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:05:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:05:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:05:18.100 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:18.100 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:18.508 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:05:18.816 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 521 kB/s | 3.1 MB 00:06 2024-03-25T18:05:18.817 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:18.818 INFO:teuthology.orchestra.run.smithi019.stdout:Total 521 kB/s | 3.1 MB 00:06 2024-03-25T18:05:18.818 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:05:18.830 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:05:18.830 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:05:18.909 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:05:18.909 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:05:19.080 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:05:32 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:19.230 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:19.232 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================== 2024-03-25T18:05:19.232 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-03-25T18:05:19.232 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================== 2024-03-25T18:05:19.232 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:05:19.232 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:05:19.232 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-03-25T18:05:19.232 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:05:19.232 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:05:19.232 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:05:19.232 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:05:19.232 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:05:19.233 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:05:19.233 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:05:19.233 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:05:19.233 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:05:19.233 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:05:19.233 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:05:19.233 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:05:19.233 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:19.233 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:05:19.233 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================== 2024-03-25T18:05:19.233 INFO:teuthology.orchestra.run.smithi188.stdout:Install 13 Packages 2024-03-25T18:05:19.233 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:19.234 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 2.0 M 2024-03-25T18:05:19.234 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 26 M 2024-03-25T18:05:19.234 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:05:19.359 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:05:19.384 INFO:teuthology.orchestra.run.smithi188.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 230 kB/s | 24 kB 00:00 2024-03-25T18:05:19.426 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:19.426 INFO:teuthology.orchestra.run.smithi188.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 122 kB/s | 18 kB 00:00 2024-03-25T18:05:19.450 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:19.450 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:19.450 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:05:19.450 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:05:19.451 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:05:19.451 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:19.476 INFO:teuthology.orchestra.run.smithi188.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T18:05:19.501 INFO:teuthology.orchestra.run.smithi188.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T18:05:19.526 INFO:teuthology.orchestra.run.smithi188.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:05:19.552 INFO:teuthology.orchestra.run.smithi188.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 868 kB/s | 43 kB 00:00 2024-03-25T18:05:19.835 INFO:teuthology.orchestra.run.smithi188.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 94 kB/s | 52 kB 00:00 2024-03-25T18:05:20.019 INFO:teuthology.orchestra.run.smithi188.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 65 kB/s | 32 kB 00:00 2024-03-25T18:05:20.044 INFO:teuthology.orchestra.run.smithi188.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 30 kB/s | 15 kB 00:00 2024-03-25T18:05:20.228 INFO:teuthology.orchestra.run.smithi188.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 324 kB/s | 59 kB 00:00 2024-03-25T18:05:20.420 INFO:teuthology.orchestra.run.smithi188.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 467 kB/s | 90 kB 00:00 2024-03-25T18:05:20.615 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:20.615 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:20.615 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:05:20.615 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:20.615 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:20.615 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:05:20.637 INFO:teuthology.orchestra.run.smithi188.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 253 kB/s | 203 kB 00:00 2024-03-25T18:05:20.831 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-nbd 2024-03-25T18:05:21.414 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:33 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:05:21.428 INFO:teuthology.orchestra.run.smithi188.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 742 kB/s | 1.0 MB 00:01 2024-03-25T18:05:21.433 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:21.433 INFO:teuthology.orchestra.run.smithi188.stdout:Total 954 kB/s | 2.0 MB 00:02 2024-03-25T18:05:21.434 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:05:21.449 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:05:21.449 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:05:21.545 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:05:21.545 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:05:21.569 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:05:21.569 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:21.570 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-03-25T18:05:21.570 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:21.570 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:05:21.570 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:05:21.570 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:21.570 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:05:21.570 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:21.570 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T18:05:21.570 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:21.570 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 171 k 2024-03-25T18:05:21.570 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 494 k 2024-03-25T18:05:21.571 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:05:22.061 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:05:22.237 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:05:22.290 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:05:22.345 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:05:22.383 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:05:22.424 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 194 kB/s | 171 kB 00:00 2024-03-25T18:05:22.453 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:22.454 INFO:teuthology.orchestra.run.smithi019.stdout:Total 194 kB/s | 171 kB 00:00 2024-03-25T18:05:22.454 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:05:22.464 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:05:22.464 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:05:22.517 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:05:22.527 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:05:22.527 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:05:22.634 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:05:22.680 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:05:22.692 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:05:22.786 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:23.211 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:23.414 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:05:23.473 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:05:23.489 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:23.489 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:23.489 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:05:23.489 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:23.489 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:23.490 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:05:23.507 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:05:23.555 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:05:23.585 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:05:23.704 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:05:24.279 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:36 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:05:24.392 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:05:24.392 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:05:24.392 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:05:24.392 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:05:24.392 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:05:24.392 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:05:24.392 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:05:24.392 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:05:24.392 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:05:24.393 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:05:24.393 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:05:24.393 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:05:24.393 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:05:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:05:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:24.434 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T18:05:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:05:24.434 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:05:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-25T18:05:24.434 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:05:24.434 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:05:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:05:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-03-25T18:05:24.435 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:05:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 91 M 2024-03-25T18:05:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 498 M 2024-03-25T18:05:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:05:24.767 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:05:24.767 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:24.767 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:05:24.767 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:24.767 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:05:24.767 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:05:24.767 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:05:24.767 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:05:24.767 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:05:24.767 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:05:24.767 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:05:24.767 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:05:24.768 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:05:24.768 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:05:24.768 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:05:24.768 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:05:24.768 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:24.768 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:25.035 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:05:25.603 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:25.754 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:25.755 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================== 2024-03-25T18:05:25.755 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-03-25T18:05:25.755 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================== 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================== 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout:Install 6 Packages 2024-03-25T18:05:25.756 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:25.757 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 6.8 M 2024-03-25T18:05:25.757 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 33 M 2024-03-25T18:05:25.757 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:05:26.250 INFO:teuthology.orchestra.run.smithi188.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-25T18:05:26.342 INFO:teuthology.orchestra.run.smithi188.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-25T18:05:26.567 INFO:teuthology.orchestra.run.smithi188.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 257 kB/s | 58 kB 00:00 2024-03-25T18:05:26.667 INFO:teuthology.orchestra.run.smithi188.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-03-25T18:05:26.700 INFO:teuthology.orchestra.run.smithi188.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 382 kB/s | 51 kB 00:00 2024-03-25T18:05:26.917 INFO:teuthology.orchestra.run.smithi188.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 160 kB/s | 138 kB 00:00 2024-03-25T18:05:26.919 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:26.920 INFO:teuthology.orchestra.run.smithi188.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2024-03-25T18:05:26.932 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:05:26.942 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:05:26.942 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:05:27.030 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:05:27.031 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:05:27.321 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:05:27.482 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:05:27.527 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:05:28.042 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:05:28.121 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:05:28.183 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:05:28.206 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:05:29.033 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:05:29.034 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:05:29.034 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:05:29.034 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:05:29.034 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:05:29.034 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:05:29.348 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:05:29.348 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:29.349 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:05:29.349 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:29.349 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:05:29.349 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:05:29.349 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:05:29.349 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:05:29.349 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:05:29.349 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:29.349 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:29.592 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-fuse 2024-03-25T18:05:30.155 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:05:43 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:30.304 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:30.305 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:30.305 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-03-25T18:05:30.305 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:30.305 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:05:30.305 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:05:30.305 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-03-25T18:05:30.305 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:05:30.305 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:30.305 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:05:30.305 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:30.305 INFO:teuthology.orchestra.run.smithi188.stdout:Install 2 Packages 2024-03-25T18:05:30.306 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:30.306 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 964 k 2024-03-25T18:05:30.306 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 2.8 M 2024-03-25T18:05:30.306 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:05:30.419 INFO:teuthology.orchestra.run.smithi188.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 694 kB/s | 79 kB 00:00 2024-03-25T18:05:33.069 INFO:teuthology.orchestra.run.smithi188.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 320 kB/s | 885 kB 00:02 2024-03-25T18:05:33.071 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:33.071 INFO:teuthology.orchestra.run.smithi188.stdout:Total 348 kB/s | 964 kB 00:02 2024-03-25T18:05:33.071 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:05:33.095 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:05:33.095 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:05:33.172 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:05:33.173 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:05:33.361 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:05:33.511 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:05:33.585 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:05:34.266 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:05:34.266 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:05:34.570 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:05:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:34.571 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:05:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:05:34.571 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:34.571 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:34.840 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-volume 2024-03-25T18:05:35.408 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:05:48 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:35.563 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:35.564 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:35.564 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-03-25T18:05:35.564 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:35.565 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:05:35.565 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:05:35.565 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-03-25T18:05:35.565 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:05:35.565 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:35.565 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:05:35.565 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:35.565 INFO:teuthology.orchestra.run.smithi188.stdout:Install 2 Packages 2024-03-25T18:05:35.565 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:35.565 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 332 k 2024-03-25T18:05:35.565 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 1.5 M 2024-03-25T18:05:35.565 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:05:35.674 INFO:teuthology.orchestra.run.smithi188.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 629 kB/s | 69 kB 00:00 2024-03-25T18:05:37.200 INFO:teuthology.orchestra.run.smithi188.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 161 kB/s | 264 kB 00:01 2024-03-25T18:05:37.202 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:37.202 INFO:teuthology.orchestra.run.smithi188.stdout:Total 203 kB/s | 332 kB 00:01 2024-03-25T18:05:37.202 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:05:37.212 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:05:37.212 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:05:37.250 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:05:37.251 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:05:37.398 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:05:37.573 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:05:37.610 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:05:37.626 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:05:37.626 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:37.626 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:05:37.626 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:38.339 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:05:38.695 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:05:38.696 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:38.696 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:05:38.696 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:38.696 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:05:38.696 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:38.696 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:38.999 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados-devel 2024-03-25T18:05:39.575 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:05:52 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:39.727 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:39.728 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:39.728 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T18:05:39.728 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:39.728 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:05:39.729 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:05:39.729 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:39.729 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:05:39.729 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:39.729 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T18:05:39.729 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:39.729 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 127 k 2024-03-25T18:05:39.729 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 456 k 2024-03-25T18:05:39.729 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:05:40.723 INFO:teuthology.orchestra.run.smithi188.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 127 kB/s | 127 kB 00:00 2024-03-25T18:05:40.724 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:40.724 INFO:teuthology.orchestra.run.smithi188.stdout:Total 127 kB/s | 127 kB 00:00 2024-03-25T18:05:40.724 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:05:40.730 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:05:40.730 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:05:40.789 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:05:40.790 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:05:40.947 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:05:41.055 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:41.533 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:41.818 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:41.823 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:41.823 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:05:41.823 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:41.823 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:41.823 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:42.041 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs2 2024-03-25T18:05:42.608 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:05:55 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:42.705 INFO:teuthology.orchestra.run.smithi188.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:42.763 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:42.764 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-03-25T18:05:42.765 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:42.826 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs-devel 2024-03-25T18:05:43.391 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:05:56 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:43.541 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:43.542 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:43.542 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T18:05:43.542 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:43.542 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:05:43.542 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:05:43.542 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:43.543 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:05:43.543 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:43.543 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T18:05:43.543 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:43.543 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 31 k 2024-03-25T18:05:43.543 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 140 k 2024-03-25T18:05:43.543 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:05:43.985 INFO:teuthology.orchestra.run.smithi188.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 71 kB/s | 31 kB 00:00 2024-03-25T18:05:43.985 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:43.985 INFO:teuthology.orchestra.run.smithi188.stdout:Total 70 kB/s | 31 kB 00:00 2024-03-25T18:05:43.986 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:05:43.989 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:05:43.989 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:05:44.020 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:05:44.020 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:05:44.134 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:05:44.233 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:44.692 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:44.961 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:44.961 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:44.961 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:05:44.961 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:44.961 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:44.961 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:45.124 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados2 2024-03-25T18:05:45.691 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:05:58 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:45.789 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:45.842 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:45.843 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-03-25T18:05:45.843 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:45.895 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librbd1 2024-03-25T18:05:46.484 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:05:59 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:46.581 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:46.634 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:46.635 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-03-25T18:05:46.636 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:46.682 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rados 2024-03-25T18:05:47.243 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:06:00 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:47.340 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:47.391 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:47.392 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-03-25T18:05:47.392 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:47.439 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rgw 2024-03-25T18:05:48.002 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:06:01 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:48.098 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:48.149 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:48.150 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-03-25T18:05:48.150 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:48.210 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-cephfs 2024-03-25T18:05:48.776 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:06:01 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:48.872 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:48.924 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:48.925 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-03-25T18:05:48.925 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:48.975 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rbd 2024-03-25T18:05:49.536 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:06:02 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:49.633 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:49.686 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:49.687 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-03-25T18:05:49.687 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:49.742 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-fuse 2024-03-25T18:05:50.306 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:50.455 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:50.455 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:50.456 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-03-25T18:05:50.456 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:50.456 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:05:50.456 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:05:50.456 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:50.456 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:05:50.456 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:50.456 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T18:05:50.456 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:50.457 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 85 k 2024-03-25T18:05:50.457 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 231 k 2024-03-25T18:05:50.457 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:05:51.371 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 93 kB/s | 85 kB 00:00 2024-03-25T18:05:51.371 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:51.372 INFO:teuthology.orchestra.run.smithi188.stdout:Total 93 kB/s | 85 kB 00:00 2024-03-25T18:05:51.372 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:05:51.380 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:05:51.380 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:05:51.437 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:05:51.438 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:05:51.577 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:05:51.686 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:52.114 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:52.393 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:52.393 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:52.393 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:05:52.393 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:52.393 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:52.393 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:05:52.655 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-mirror 2024-03-25T18:05:53.227 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:06:06 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:05:53.379 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:05:53.380 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:53.380 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T18:05:53.380 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:53.381 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:05:53.381 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:05:53.381 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:53.381 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:05:53.381 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:05:53.381 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T18:05:53.381 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:05:53.381 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 3.1 M 2024-03-25T18:05:53.381 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 13 M 2024-03-25T18:05:53.381 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:06:00.907 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 420 kB/s | 3.1 MB 00:07 2024-03-25T18:06:00.907 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:00.908 INFO:teuthology.orchestra.run.smithi188.stdout:Total 420 kB/s | 3.1 MB 00:07 2024-03-25T18:06:00.908 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:06:00.919 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:06:00.919 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:06:00.996 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:06:00.996 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:06:01.429 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:06:01.491 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:01.515 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:01.516 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:01.516 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:06:01.516 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:06:01.516 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:06:01.516 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:06:02.606 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:02.606 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:06:02.607 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:06:02.607 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:02.607 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:06:02.607 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:06:02.845 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-nbd 2024-03-25T18:06:03.424 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:06:16 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:06:03.573 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:06:03.574 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:06:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-03-25T18:06:03.574 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:06:03.574 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:06:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:06:03.574 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:06:03.574 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:06:03.574 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:06:03.574 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T18:06:03.575 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:06:03.575 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 171 k 2024-03-25T18:06:03.575 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 494 k 2024-03-25T18:06:03.575 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:06:05.685 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 81 kB/s | 171 kB 00:02 2024-03-25T18:06:05.686 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:05.687 INFO:teuthology.orchestra.run.smithi188.stdout:Total 81 kB/s | 171 kB 00:02 2024-03-25T18:06:05.687 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:06:05.696 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:06:05.696 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:06:05.756 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:06:05.757 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:06:05.913 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:06:06.017 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:06.467 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:06.789 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:06.789 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:06:06.789 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:06:06.789 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:06.789 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:06:06.789 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:06:07.047 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:06:07.624 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:06:20 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:06:07.780 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout:Install 2 Packages 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 91 M 2024-03-25T18:06:07.781 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 498 M 2024-03-25T18:06:07.782 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:06:24.267 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 384 kB/s | 22 MB 00:59 2024-03-25T18:07:10.119 INFO:teuthology.orchestra.run.smithi188.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 369 kB/s | 22 MB 01:02 2024-03-25T18:09:14.627 INFO:teuthology.orchestra.run.smithi188.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 373 kB/s | 68 MB 03:06 2024-03-25T18:09:14.628 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:14.628 INFO:teuthology.orchestra.run.smithi188.stdout:Total 496 kB/s | 91 MB 03:06 2024-03-25T18:09:14.629 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:09:14.632 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:09:14.632 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:09:15.225 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:09:15.225 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:09:18.376 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:09:25.562 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:09:25.736 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:09:29.196 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:09:29.197 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:09:29.458 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:09:29.458 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:09:29.458 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:09:29.458 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:29.458 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:29.458 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:09:29.459 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:09:29.668 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:09:30.235 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:09:43 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:09:30.384 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:09:30.384 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:09:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T18:09:30.384 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:09:30.384 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:09:30.384 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:09:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:09:30.385 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:09:30.385 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:09:30.385 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T18:09:30.385 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:09:30.385 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 416 M 2024-03-25T18:09:30.386 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 1.7 G 2024-03-25T18:09:30.386 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:09:33.537 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 280 kB/s | 68 MB 04:09 2024-03-25T18:09:33.539 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:33.539 INFO:teuthology.orchestra.run.smithi019.stdout:Total 372 kB/s | 91 MB 04:09 2024-03-25T18:09:33.539 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:09:33.543 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:09:33.543 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:09:34.150 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:09:34.150 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:09:37.427 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:09:44.931 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:09:44.999 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:09:48.360 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:09:48.360 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:09:48.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:09:48.630 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:09:48.630 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:09:48.630 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:48.630 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:48.630 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:09:48.631 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:09:48.837 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:09:49.415 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:10:01 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:09:49.571 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:09:49.572 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:09:49.572 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T18:09:49.572 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:09:49.572 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:09:49.572 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:09:49.572 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:09:49.572 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:09:49.572 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:09:49.572 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T18:09:49.572 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:09:49.572 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 416 M 2024-03-25T18:09:49.572 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.7 G 2024-03-25T18:09:49.573 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:22:29.327 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 561 kB/s | 416 MB 12:39 2024-03-25T18:22:29.328 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:29.328 INFO:teuthology.orchestra.run.smithi019.stdout:Total 561 kB/s | 416 MB 12:39 2024-03-25T18:22:29.328 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:22:29.330 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:22:29.331 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:22:31.920 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:22:31.920 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:23:16.580 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:23:16.691 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:24.686 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:24.955 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:24.955 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:23:24.955 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:23:24.955 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:24.955 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:23:24.955 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:23:25.187 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:23:25.762 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:23:37 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:23:25.918 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:23:25.919 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-25T18:23:25.919 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T18:23:25.919 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-25T18:23:25.919 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:23:25.920 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:23:25.920 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:23:25.920 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:23:25.920 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-25T18:23:25.920 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T18:23:25.920 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:23:25.920 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 1.4 M 2024-03-25T18:23:25.920 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 6.4 M 2024-03-25T18:23:25.920 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:23:28.558 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 554 kB/s | 1.4 MB 00:02 2024-03-25T18:23:28.559 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:28.560 INFO:teuthology.orchestra.run.smithi019.stdout:Total 553 kB/s | 1.4 MB 00:02 2024-03-25T18:23:28.560 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:23:28.564 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:23:28.564 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:23:28.611 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:23:28.611 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:23:28.877 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:23:28.985 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:23:29.501 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:23:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:23:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:23:29.793 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:23:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:29.793 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:23:29.793 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:23:29.972 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:23:30.543 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:23:42 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:23:30.697 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:23:30.698 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:23:30.698 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T18:23:30.698 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:23:30.698 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:23:30.699 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:23:30.699 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:23:30.699 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:23:30.699 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:23:30.699 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T18:23:30.699 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:23:30.699 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 176 M 2024-03-25T18:23:30.699 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 753 M 2024-03-25T18:23:30.699 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:25:41.656 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 439 kB/s | 416 MB 16:11 2024-03-25T18:25:41.656 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:41.656 INFO:teuthology.orchestra.run.smithi188.stdout:Total 439 kB/s | 416 MB 16:11 2024-03-25T18:25:41.656 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:25:41.659 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:25:41.659 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:25:44.125 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:25:44.125 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:26:26.457 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:26:26.564 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:31.659 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:31.914 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:31.915 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:26:31.915 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:26:31.915 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:31.915 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:26:31.915 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:26:32.152 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:26:32.714 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:26:45 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:26:32.861 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:26:32.862 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================== 2024-03-25T18:26:32.862 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T18:26:32.862 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================== 2024-03-25T18:26:32.862 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:26:32.863 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:26:32.863 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:26:32.863 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:26:32.863 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================== 2024-03-25T18:26:32.863 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T18:26:32.863 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:26:32.863 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 1.4 M 2024-03-25T18:26:32.863 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 6.4 M 2024-03-25T18:26:32.863 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:26:37.465 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 317 kB/s | 1.4 MB 00:04 2024-03-25T18:26:37.469 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:26:37.469 INFO:teuthology.orchestra.run.smithi188.stdout:Total 317 kB/s | 1.4 MB 00:04 2024-03-25T18:26:37.469 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:26:37.474 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:26:37.474 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:26:37.519 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:26:37.519 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:26:37.795 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:26:37.897 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:38.315 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:38.594 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:38.594 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:26:38.594 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:26:38.594 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:38.595 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:26:38.595 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:26:38.774 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:26:39.330 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:26:52 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:26:39.478 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 176 M 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 753 M 2024-03-25T18:26:39.479 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T18:29:11.042 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 529 kB/s | 176 MB 05:40 2024-03-25T18:29:11.043 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:29:11.043 INFO:teuthology.orchestra.run.smithi019.stdout:Total 529 kB/s | 176 MB 05:40 2024-03-25T18:29:11.043 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T18:29:11.046 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T18:29:11.046 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T18:29:12.156 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T18:29:12.156 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T18:29:31.188 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T18:29:31.291 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:29:35.827 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:29:36.103 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:29:36.103 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:29:36.103 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T18:29:36.103 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:29:36.104 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:29:36.104 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T18:29:36.399 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:29:36.975 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:29:48 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T18:29:37.128 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T18:29:37.129 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:29:37.129 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T18:29:37.129 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:29:37.130 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T18:29:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:29:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:29:37.130 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T18:29:37.130 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T18:29:37.130 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T18:29:37.130 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T18:29:37.130 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 672 M 2024-03-25T18:29:37.130 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 3.1 G 2024-03-25T18:29:37.130 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T18:32:53.968 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 480 kB/s | 176 MB 06:14 2024-03-25T18:32:53.968 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:32:53.968 INFO:teuthology.orchestra.run.smithi188.stdout:Total 480 kB/s | 176 MB 06:14 2024-03-25T18:32:53.969 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T18:32:53.971 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T18:32:53.971 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T18:32:55.031 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T18:32:55.032 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T18:33:13.011 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T18:33:13.195 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:33:18.049 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:33:18.315 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:33:18.315 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:33:18.315 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T18:33:18.315 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:18.315 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:33:18.315 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T18:33:18.544 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:33:19.109 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:33:32 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T18:33:19.258 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T18:33:19.258 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:33:19.258 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T18:33:19.259 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:33:19.259 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T18:33:19.259 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:33:19.259 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:33:19.259 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T18:33:19.259 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T18:33:19.259 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T18:33:19.259 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T18:33:19.260 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 672 M 2024-03-25T18:33:19.260 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 3.1 G 2024-03-25T18:33:19.260 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:00:11.638 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 427 kB/s | 672 MB 26:52 2024-03-25T19:00:11.639 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:00:11.639 INFO:teuthology.orchestra.run.smithi188.stdout:Total 427 kB/s | 672 MB 26:52 2024-03-25T19:00:11.639 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:00:11.644 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:00:11.644 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:00:15.598 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:00:15.598 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:01:26.490 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:01:27.061 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:01:34.881 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:01:35.163 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:01:35.163 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:01:35.163 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:01:35.163 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:01:35.163 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:01:35.163 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:01:35.441 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:01:36.009 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:01:49 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:01:36.105 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:01:36.157 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:01:36.158 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-03-25T19:01:36.158 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:01:36.229 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:01:36.796 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:01:49 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:01:36.944 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:01:36.945 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:01:36.945 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:01:36.945 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:01:36.945 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:01:36.945 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:01:36.945 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:01:36.945 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:01:36.945 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:01:36.945 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:01:36.945 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:01:36.945 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 20 M 2024-03-25T19:01:36.946 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 103 M 2024-03-25T19:01:36.946 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:01:37.119 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 359 kB/s | 672 MB 31:59 2024-03-25T19:01:37.119 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:01:37.120 INFO:teuthology.orchestra.run.smithi019.stdout:Total 359 kB/s | 672 MB 31:59 2024-03-25T19:01:37.120 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:01:37.125 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:01:37.125 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:01:41.279 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:01:41.279 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:02:26.110 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 410 kB/s | 20 MB 00:49 2024-03-25T19:02:26.111 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:02:26.112 INFO:teuthology.orchestra.run.smithi188.stdout:Total 410 kB/s | 20 MB 00:49 2024-03-25T19:02:26.112 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:02:26.116 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:02:26.116 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:02:26.265 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:02:26.265 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:02:28.485 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:02:28.599 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:02:29.629 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:02:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:02:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:02:29.899 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:02:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:02:29.899 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:02:29.899 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:02:30.074 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:02:30.636 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:02:43 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:02:30.785 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:02:30.786 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:02:30.786 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:02:30.786 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:02:30.786 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:02:30.786 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:02:30.786 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:02:30.786 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:02:30.787 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:02:30.787 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:02:30.787 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:02:30.787 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 48 M 2024-03-25T19:02:30.787 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 173 M 2024-03-25T19:02:30.787 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:02:56.228 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:02:56.349 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:03:04.032 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:03:04.307 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:03:04.307 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:03:04.307 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:03:04.307 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:03:04.307 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:03:04.307 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:03:04.597 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:03:05.172 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:03:17 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:03:05.270 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:03:05.325 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:03:05.326 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-25T19:03:05.326 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:03:05.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:03:05.964 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:03:17 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:03:06.121 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:03:06.121 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:03:06.121 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:03:06.121 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:03:06.122 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:03:06.122 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:03:06.122 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:03:06.122 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:03:06.122 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:03:06.122 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:03:06.122 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:03:06.122 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 20 M 2024-03-25T19:03:06.122 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 103 M 2024-03-25T19:03:06.122 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:03:59.170 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 380 kB/s | 20 MB 00:53 2024-03-25T19:03:59.170 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:03:59.171 INFO:teuthology.orchestra.run.smithi019.stdout:Total 380 kB/s | 20 MB 00:53 2024-03-25T19:03:59.171 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:03:59.174 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:03:59.174 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:03:59.330 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:03:59.330 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:04:01.683 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:04:01.787 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:04:02.748 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:04:03.022 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:04:03.022 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:04:03.022 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:04:03.022 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:03.022 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:04:03.022 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:04:03.198 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:04:03.774 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:04:15 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:04:03.929 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:04:03.930 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:04:03.930 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:04:03.930 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:04:03.930 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:04:03.930 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:04:03.930 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:04:03.930 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:04:03.931 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:04:03.931 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:04:03.931 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:04:03.931 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 48 M 2024-03-25T19:04:03.931 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 173 M 2024-03-25T19:04:03.931 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:04:23.056 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 435 kB/s | 48 MB 01:52 2024-03-25T19:04:23.057 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:23.057 INFO:teuthology.orchestra.run.smithi188.stdout:Total 435 kB/s | 48 MB 01:52 2024-03-25T19:04:23.058 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:04:23.061 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:04:23.061 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:04:23.373 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:04:23.373 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:04:28.129 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:04:28.255 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:04:29.680 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:04:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:04:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:04:29.954 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:04:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:29.954 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:04:29.954 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:04:30.129 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:04:30.689 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:04:43 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:04:30.838 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:04:30.839 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:04:30.839 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:04:30.839 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:04:30.839 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:04:30.839 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:04:30.839 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:04:30.839 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:04:30.839 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:04:30.839 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:04:30.839 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:04:30.840 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 99 M 2024-03-25T19:04:30.840 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 449 M 2024-03-25T19:04:30.840 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:06:04.801 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 404 kB/s | 48 MB 02:00 2024-03-25T19:06:04.801 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:06:04.801 INFO:teuthology.orchestra.run.smithi019.stdout:Total 404 kB/s | 48 MB 02:00 2024-03-25T19:06:04.802 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:06:04.805 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:06:04.805 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:06:05.133 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:06:05.133 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:06:10.146 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:06:10.235 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:06:11.648 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:06:11.930 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:06:11.930 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:06:11.930 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:06:11.930 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:06:11.930 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:06:11.930 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:06:12.107 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:06:12.680 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:06:24 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:06:12.835 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:06:12.835 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:06:12.835 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:06:12.835 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:06:12.835 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:06:12.835 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:06:12.836 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:06:12.836 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:06:12.836 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:06:12.836 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:06:12.836 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:06:12.836 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 99 M 2024-03-25T19:06:12.836 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 449 M 2024-03-25T19:06:12.836 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:07:29.698 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 564 kB/s | 99 MB 02:58 2024-03-25T19:07:29.698 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:07:29.699 INFO:teuthology.orchestra.run.smithi188.stdout:Total 564 kB/s | 99 MB 02:58 2024-03-25T19:07:29.699 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:07:29.701 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:07:29.702 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:07:30.309 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:07:30.310 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:07:40.550 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:07:40.643 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:43.539 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:43.802 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:43.802 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:07:43.802 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:07:43.803 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:07:43.803 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:07:43.803 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:07:43.998 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:07:44.561 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:07:57 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:07:44.709 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:07:44.709 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:07:44.709 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:07:44.709 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:07:44.709 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:07:44.709 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:07:44.709 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:07:44.710 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:07:44.710 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:07:44.710 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:07:44.710 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:07:44.710 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 287 M 2024-03-25T19:07:44.710 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 1.2 G 2024-03-25T19:07:44.710 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:10:56.481 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 356 kB/s | 99 MB 04:43 2024-03-25T19:10:56.481 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:10:56.481 INFO:teuthology.orchestra.run.smithi019.stdout:Total 356 kB/s | 99 MB 04:43 2024-03-25T19:10:56.481 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:10:56.484 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:10:56.484 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:10:57.122 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:10:57.122 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:11:08.022 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:11:08.113 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:11:11.115 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:11:11.389 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:11:11.389 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:11:11.389 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:11:11.389 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:11:11.389 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:11:11.389 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:11:11.550 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:11:12.126 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:11:24 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:11:12.280 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:11:12.281 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:11:12.281 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:11:12.281 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:11:12.281 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:11:12.281 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:11:12.281 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:11:12.281 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:11:12.281 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:11:12.281 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:11:12.281 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:11:12.282 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 287 M 2024-03-25T19:11:12.282 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.2 G 2024-03-25T19:11:12.282 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:19:09.122 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 429 kB/s | 287 MB 11:24 2024-03-25T19:19:09.122 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:09.123 INFO:teuthology.orchestra.run.smithi188.stdout:Total 429 kB/s | 287 MB 11:24 2024-03-25T19:19:09.123 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:19:09.125 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:19:09.125 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:19:10.828 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:19:10.828 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:19:40.189 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:19:40.290 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:19:41.468 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:19:41.777 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:19:41.777 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:19:41.777 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:19:41.777 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:41.777 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:19:41.777 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:19:41.971 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:19:42.536 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:19:55 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:19:42.684 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:19:42.685 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:19:42.685 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:19:42.685 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:19:42.685 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:19:42.685 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:19:42.685 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:19:42.685 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:19:42.685 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:19:42.685 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:19:42.685 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:19:42.686 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 5.9 M 2024-03-25T19:19:42.686 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 28 M 2024-03-25T19:19:42.686 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:19:59.749 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 357 kB/s | 5.9 MB 00:17 2024-03-25T19:19:59.750 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:59.750 INFO:teuthology.orchestra.run.smithi188.stdout:Total 357 kB/s | 5.9 MB 00:17 2024-03-25T19:19:59.751 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:19:59.755 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:19:59.755 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:19:59.827 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:19:59.827 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:20:00.568 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:20:00.663 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:20:01.260 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:20:01.538 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:20:01.538 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:20:01.539 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:20:01.539 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:20:01.539 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:20:01.539 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:20:01.759 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:20:02.327 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:20:15 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:20:02.480 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:20:02.480 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:20:02.480 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:20:02.481 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:20:02.481 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:20:02.481 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:20:02.481 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:20:02.481 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:20:02.481 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:20:02.481 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:20:02.481 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:20:02.481 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 109 k 2024-03-25T19:20:02.481 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 440 k 2024-03-25T19:20:02.481 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:20:03.044 INFO:teuthology.orchestra.run.smithi188.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 193 kB/s | 109 kB 00:00 2024-03-25T19:20:03.044 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:20:03.045 INFO:teuthology.orchestra.run.smithi188.stdout:Total 193 kB/s | 109 kB 00:00 2024-03-25T19:20:03.045 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:20:03.048 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:20:03.048 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:20:03.083 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:20:03.083 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:20:03.203 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:20:03.317 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:20:03.691 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:20:03.972 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:20:03.972 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:20:03.972 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:20:03.972 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:20:03.973 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:20:03.973 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:20:04.165 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:20:04.733 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:20:17 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:20:04.884 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:20:04.884 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:20:04.885 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:20:04.885 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:20:04.885 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:20:04.885 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:20:04.885 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:20:04.885 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:20:04.885 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:20:04.885 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:20:04.885 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:20:04.885 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 5.4 M 2024-03-25T19:20:04.885 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 25 M 2024-03-25T19:20:04.885 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:20:12.664 INFO:teuthology.orchestra.run.smithi188.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 705 kB/s | 5.4 MB 00:07 2024-03-25T19:20:12.665 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:20:12.665 INFO:teuthology.orchestra.run.smithi188.stdout:Total 704 kB/s | 5.4 MB 00:07 2024-03-25T19:20:12.666 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:20:12.670 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:20:12.670 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:20:12.738 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:20:12.738 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:20:13.438 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:20:13.546 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:20:14.111 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:20:14.368 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:20:14.368 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:20:14.368 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:20:14.368 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:20:14.368 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:20:14.368 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:20:14.573 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados2-debuginfo 2024-03-25T19:20:15.139 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:20:28 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:20:15.289 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:20:15.289 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:20:15.289 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:20:15.289 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:20:15.289 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:20:15.290 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:20:15.290 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:20:15.290 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:20:15.290 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:20:15.290 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:20:15.290 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:20:15.291 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 44 M 2024-03-25T19:20:15.291 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 253 M 2024-03-25T19:20:15.291 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:22:07.070 INFO:teuthology.orchestra.run.smithi188.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 406 kB/s | 44 MB 01:51 2024-03-25T19:22:07.070 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:07.071 INFO:teuthology.orchestra.run.smithi188.stdout:Total 406 kB/s | 44 MB 01:51 2024-03-25T19:22:07.071 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:22:07.073 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:22:07.073 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:22:07.372 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:22:07.372 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:22:12.417 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:22:12.516 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:14.418 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:14.674 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:14.674 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:22:14.674 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:22:14.674 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:22:14.675 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:22:14.888 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:22:15.456 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:22:28 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:22:15.606 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:22:15.606 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:22:15.606 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:22:15.606 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:22:15.606 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:22:15.606 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:22:15.606 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:22:15.606 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:22:15.606 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:22:15.606 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:22:15.607 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:22:15.607 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 52 M 2024-03-25T19:22:15.607 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 308 M 2024-03-25T19:22:15.607 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:22:25.899 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 436 kB/s | 287 MB 11:13 2024-03-25T19:22:25.900 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:25.900 INFO:teuthology.orchestra.run.smithi019.stdout:Total 436 kB/s | 287 MB 11:13 2024-03-25T19:22:25.900 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:22:25.904 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:22:25.904 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:22:27.694 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:22:27.695 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:22:58.854 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:22:58.957 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:06.401 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:06.718 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:06.718 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:06.718 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:23:06.719 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:06.719 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:06.719 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:23:06.933 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:23:07.507 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:23:19 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:23:07.662 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:23:07.663 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:23:07.663 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:23:07.663 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:23:07.663 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:23:07.663 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:23:07.663 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:07.664 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:23:07.664 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:23:07.664 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:23:07.664 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:07.664 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 5.9 M 2024-03-25T19:23:07.664 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 28 M 2024-03-25T19:23:07.664 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:23:16.999 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 652 kB/s | 5.9 MB 00:09 2024-03-25T19:23:16.999 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:17.000 INFO:teuthology.orchestra.run.smithi019.stdout:Total 652 kB/s | 5.9 MB 00:09 2024-03-25T19:23:17.000 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:23:17.004 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:23:17.004 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:23:17.080 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:23:17.080 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:23:17.849 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:23:17.950 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:23:18.557 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:23:18.835 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:23:18.836 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:18.836 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:23:18.836 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:18.836 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:18.836 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:23:19.040 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:23:19.612 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:23:31 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:23:19.770 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:23:19.771 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:23:19.771 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:23:19.771 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:23:19.771 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:23:19.772 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:23:19.772 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:19.772 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:23:19.772 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:23:19.772 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:23:19.772 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:19.772 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 109 k 2024-03-25T19:23:19.772 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 440 k 2024-03-25T19:23:19.772 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:23:20.485 INFO:teuthology.orchestra.run.smithi019.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 152 kB/s | 109 kB 00:00 2024-03-25T19:23:20.486 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:20.486 INFO:teuthology.orchestra.run.smithi019.stdout:Total 152 kB/s | 109 kB 00:00 2024-03-25T19:23:20.486 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:23:20.490 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:23:20.490 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:23:20.528 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:23:20.528 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:23:20.649 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:23:20.786 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:23:21.177 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:23:21.450 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:23:21.451 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:21.451 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:23:21.451 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:21.451 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:21.451 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:23:21.742 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:23:22.319 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:23:34 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:23:22.476 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:23:22.477 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:23:22.477 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:23:22.477 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:23:22.477 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:23:22.477 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:23:22.477 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:22.478 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:23:22.478 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:23:22.478 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:23:22.478 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:22.478 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 5.4 M 2024-03-25T19:23:22.478 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 25 M 2024-03-25T19:23:22.478 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:23:31.804 INFO:teuthology.orchestra.run.smithi019.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 588 kB/s | 5.4 MB 00:09 2024-03-25T19:23:31.805 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:31.805 INFO:teuthology.orchestra.run.smithi019.stdout:Total 588 kB/s | 5.4 MB 00:09 2024-03-25T19:23:31.806 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:23:31.810 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:23:31.810 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:23:31.881 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:23:31.881 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:23:32.585 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:23:32.697 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:23:33.213 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:23:33.525 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:23:33.526 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:33.526 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:23:33.526 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:33.526 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:33.526 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:23:33.722 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados2-debuginfo 2024-03-25T19:23:34.299 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:23:46 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:23:34.454 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 44 M 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 253 M 2024-03-25T19:23:34.455 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:23:49.443 INFO:teuthology.orchestra.run.smithi188.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 566 kB/s | 52 MB 01:33 2024-03-25T19:23:49.444 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:49.444 INFO:teuthology.orchestra.run.smithi188.stdout:Total 566 kB/s | 52 MB 01:33 2024-03-25T19:23:49.444 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:23:49.448 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:23:49.448 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:23:49.793 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:23:49.793 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:23:55.792 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:23:55.933 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:23:58.150 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:23:58.426 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:23:58.426 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:23:58.426 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:23:58.426 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:58.426 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:23:58.426 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:23:58.632 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:23:59.188 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:24:12 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:23:59.336 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:23:59.337 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:23:59.337 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:23:59.337 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:23:59.337 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:23:59.337 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:23:59.337 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:23:59.337 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:23:59.337 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:23:59.337 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:23:59.337 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:23:59.337 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 615 k 2024-03-25T19:23:59.337 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 2.1 M 2024-03-25T19:23:59.338 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:24:02.761 INFO:teuthology.orchestra.run.smithi188.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 180 kB/s | 615 kB 00:03 2024-03-25T19:24:02.762 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:02.762 INFO:teuthology.orchestra.run.smithi188.stdout:Total 180 kB/s | 615 kB 00:03 2024-03-25T19:24:02.763 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:24:02.766 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:24:02.766 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:24:02.809 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:24:02.809 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:24:03.025 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:24:03.118 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:24:03.522 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:24:03.817 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:24:03.818 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:03.818 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:24:03.818 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:03.818 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:03.818 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:24:04.028 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:24:04.589 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:24:17 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:24:04.741 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:24:04.742 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:04.742 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:24:04.742 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:04.742 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:24:04.742 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:24:04.742 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:04.742 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:24:04.742 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:04.742 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:24:04.742 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:04.743 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 771 k 2024-03-25T19:24:04.743 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 2.7 M 2024-03-25T19:24:04.743 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:24:07.556 INFO:teuthology.orchestra.run.smithi188.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 274 kB/s | 771 kB 00:02 2024-03-25T19:24:07.556 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:07.557 INFO:teuthology.orchestra.run.smithi188.stdout:Total 274 kB/s | 771 kB 00:02 2024-03-25T19:24:07.557 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:24:07.560 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:24:07.561 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:24:07.602 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:24:07.603 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:24:07.789 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:24:07.904 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:24:08.298 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:24:08.562 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:24:08.563 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:08.563 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:24:08.563 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:08.563 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:08.563 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:24:08.791 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:24:09.357 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:24:22 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:24:09.506 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:24:09.507 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:09.507 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:24:09.507 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:09.507 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:24:09.507 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:24:09.507 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:09.507 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:24:09.508 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:09.508 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:24:09.508 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:09.508 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 1.1 M 2024-03-25T19:24:09.508 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 4.3 M 2024-03-25T19:24:09.508 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:24:12.033 INFO:teuthology.orchestra.run.smithi188.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 456 kB/s | 1.1 MB 00:02 2024-03-25T19:24:12.034 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:12.034 INFO:teuthology.orchestra.run.smithi188.stdout:Total 455 kB/s | 1.1 MB 00:02 2024-03-25T19:24:12.034 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:24:12.038 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:24:12.039 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:24:12.084 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:24:12.084 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:24:12.311 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:24:12.407 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:24:12.768 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:24:13.040 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:24:13.041 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:13.041 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:24:13.041 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:13.041 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:13.041 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:24:13.252 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:24:13.815 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:24:26 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:24:13.963 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:24:13.964 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:13.964 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:24:13.964 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:13.964 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:24:13.964 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:24:13.964 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:13.964 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:24:13.964 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:13.964 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:24:13.965 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:13.965 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 337 k 2024-03-25T19:24:13.965 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 1.0 M 2024-03-25T19:24:13.965 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:24:15.111 INFO:teuthology.orchestra.run.smithi188.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 294 kB/s | 337 kB 00:01 2024-03-25T19:24:15.111 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:15.112 INFO:teuthology.orchestra.run.smithi188.stdout:Total 294 kB/s | 337 kB 00:01 2024-03-25T19:24:15.112 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:24:15.116 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:24:15.116 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:24:15.153 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:24:15.153 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:24:15.294 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:24:15.419 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:24:15.839 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:24:16.106 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:24:16.106 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:16.106 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:24:16.106 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:16.106 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:16.106 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:24:16.325 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:24:16.886 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:24:29 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:24:17.036 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:24:17.037 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:17.037 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:24:17.037 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:17.037 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:24:17.037 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:24:17.037 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:17.037 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:24:17.037 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:17.037 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:24:17.037 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:17.037 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 509 k 2024-03-25T19:24:17.038 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 2.5 M 2024-03-25T19:24:17.038 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:24:18.797 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 289 kB/s | 509 kB 00:01 2024-03-25T19:24:18.797 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:18.798 INFO:teuthology.orchestra.run.smithi188.stdout:Total 289 kB/s | 509 kB 00:01 2024-03-25T19:24:18.798 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:24:18.802 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:24:18.802 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:24:18.843 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:24:18.843 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:24:19.007 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:24:19.104 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:19.477 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:19.764 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:19.764 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:19.764 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:24:19.764 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:19.764 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:19.764 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:24:19.962 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:24:20.528 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:24:33 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:24:20.677 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:24:20.678 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:20.678 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:24:20.678 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:20.679 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:24:20.679 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:24:20.679 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:20.679 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:24:20.679 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:24:20.679 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:24:20.679 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:24:20.679 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 52 M 2024-03-25T19:24:20.679 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 298 M 2024-03-25T19:24:20.679 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:24:22.661 INFO:teuthology.orchestra.run.smithi019.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 941 kB/s | 44 MB 00:48 2024-03-25T19:24:22.662 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:22.662 INFO:teuthology.orchestra.run.smithi019.stdout:Total 941 kB/s | 44 MB 00:48 2024-03-25T19:24:22.662 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:24:22.666 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:24:22.666 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:24:22.979 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:24:22.980 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:24:28.327 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:24:28.437 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:30.245 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:30.515 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:30.516 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:24:30.516 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:24:30.516 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:30.516 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:24:30.516 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:24:30.708 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:24:31.283 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:24:43 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:24:31.440 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:24:31.441 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:24:31.441 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:24:31.441 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:24:31.441 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:24:31.441 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:24:31.441 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:24:31.441 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:24:31.441 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:24:31.441 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:24:31.441 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:24:31.441 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 52 M 2024-03-25T19:24:31.441 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 308 M 2024-03-25T19:24:31.442 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:25:33.802 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 722 kB/s | 52 MB 01:13 2024-03-25T19:25:33.802 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:33.803 INFO:teuthology.orchestra.run.smithi188.stdout:Total 722 kB/s | 52 MB 01:13 2024-03-25T19:25:33.803 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:25:33.807 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:25:33.807 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:25:34.142 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:25:34.142 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:25:40.065 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:25:40.291 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:25:42.402 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:25:42.677 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:25:42.677 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:25:42.677 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:25:42.677 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:42.677 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:25:42.678 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:25:42.883 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:25:43.447 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 1:25:56 ago on Mon 25 Mar 2024 05:59:47 PM UTC. 2024-03-25T19:25:43.598 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-03-25T19:25:43.599 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:25:43.599 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-03-25T19:25:43.599 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:25:43.599 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-03-25T19:25:43.599 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:25:43.599 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:25:43.599 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-03-25T19:25:43.599 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-03-25T19:25:43.599 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-03-25T19:25:43.599 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:25:43.600 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 1.3 M 2024-03-25T19:25:43.600 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 5.5 M 2024-03-25T19:25:43.600 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-03-25T19:25:46.969 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 381 kB/s | 1.3 MB 00:03 2024-03-25T19:25:46.970 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:46.970 INFO:teuthology.orchestra.run.smithi188.stdout:Total 380 kB/s | 1.3 MB 00:03 2024-03-25T19:25:46.970 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-03-25T19:25:46.974 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-03-25T19:25:46.974 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-03-25T19:25:47.019 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-03-25T19:25:47.019 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-03-25T19:25:47.276 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-03-25T19:25:47.392 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:25:47.819 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:25:48.092 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:25:48.092 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:25:48.092 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-03-25T19:25:48.092 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:48.092 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:25:48.093 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-03-25T19:25:48.294 DEBUG:teuthology.parallel:result is None 2024-03-25T19:25:54.355 INFO:teuthology.orchestra.run.smithi019.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 640 kB/s | 52 MB 01:22 2024-03-25T19:25:54.356 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:54.356 INFO:teuthology.orchestra.run.smithi019.stdout:Total 640 kB/s | 52 MB 01:22 2024-03-25T19:25:54.356 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:25:54.360 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:25:54.360 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:25:54.721 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:25:54.722 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:26:01.082 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:26:01.190 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:26:03.369 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:26:03.646 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:26:03.646 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:03.646 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:26:03.647 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:03.647 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:03.647 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:26:03.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:26:04.420 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:26:16 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:26:04.576 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:26:04.576 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:04.576 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:26:04.576 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:04.577 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:26:04.577 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:26:04.577 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:04.577 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:26:04.577 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:04.577 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:26:04.577 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:04.577 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 615 k 2024-03-25T19:26:04.578 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 2.1 M 2024-03-25T19:26:04.578 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:26:05.775 INFO:teuthology.orchestra.run.smithi019.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 514 kB/s | 615 kB 00:01 2024-03-25T19:26:05.775 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:05.776 INFO:teuthology.orchestra.run.smithi019.stdout:Total 513 kB/s | 615 kB 00:01 2024-03-25T19:26:05.776 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:26:05.780 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:26:05.780 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:26:05.824 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:26:05.824 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:26:05.998 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:26:06.118 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:26:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:26:06.828 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:26:06.828 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:06.828 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:26:06.829 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:06.829 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:06.829 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:26:07.014 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:26:07.590 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:26:19 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:26:07.750 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 771 k 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 2.7 M 2024-03-25T19:26:07.751 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:26:11.903 INFO:teuthology.orchestra.run.smithi019.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 186 kB/s | 771 kB 00:04 2024-03-25T19:26:11.904 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:11.904 INFO:teuthology.orchestra.run.smithi019.stdout:Total 186 kB/s | 771 kB 00:04 2024-03-25T19:26:11.904 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:26:11.908 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:26:11.908 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:26:11.953 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:26:11.953 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:26:12.146 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:26:12.254 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:26:12.645 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:26:12.912 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:26:12.912 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:12.912 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:26:12.912 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:12.912 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:12.912 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:26:13.118 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:26:13.693 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:26:25 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:26:13.847 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:26:13.848 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:13.848 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:26:13.848 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:13.848 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:26:13.849 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:26:13.849 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:13.849 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:26:13.849 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:13.849 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:26:13.849 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:13.849 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 1.1 M 2024-03-25T19:26:13.849 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 4.3 M 2024-03-25T19:26:13.849 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:26:15.431 INFO:teuthology.orchestra.run.smithi019.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 728 kB/s | 1.1 MB 00:01 2024-03-25T19:26:15.431 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:15.432 INFO:teuthology.orchestra.run.smithi019.stdout:Total 727 kB/s | 1.1 MB 00:01 2024-03-25T19:26:15.432 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:26:15.436 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:26:15.436 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:26:15.482 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:26:15.483 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:26:15.713 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:26:15.817 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:26:16.218 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:26:16.487 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:26:16.487 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:16.487 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:26:16.487 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:16.487 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:16.487 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:26:16.683 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:26:17.257 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:26:29 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:26:17.414 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:26:17.415 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:17.415 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:26:17.415 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:17.415 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:26:17.415 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:26:17.415 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:17.415 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:26:17.415 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:17.415 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:26:17.415 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:17.415 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 337 k 2024-03-25T19:26:17.416 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.0 M 2024-03-25T19:26:17.416 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:26:18.276 INFO:teuthology.orchestra.run.smithi019.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 392 kB/s | 337 kB 00:00 2024-03-25T19:26:18.277 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:18.277 INFO:teuthology.orchestra.run.smithi019.stdout:Total 392 kB/s | 337 kB 00:00 2024-03-25T19:26:18.277 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:26:18.281 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:26:18.281 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:26:18.321 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:26:18.321 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:26:18.467 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:26:18.611 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:26:18.979 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:26:19.252 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:26:19.252 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:19.252 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:26:19.252 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:19.252 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:19.253 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:26:19.449 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:26:20.020 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:26:32 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:26:20.185 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:26:20.186 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:20.186 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:26:20.186 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:20.186 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:26:20.186 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:26:20.186 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:20.186 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:26:20.186 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:20.187 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:26:20.187 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:20.187 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 509 k 2024-03-25T19:26:20.187 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 2.5 M 2024-03-25T19:26:20.187 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:26:22.446 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 225 kB/s | 509 kB 00:02 2024-03-25T19:26:22.447 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:22.448 INFO:teuthology.orchestra.run.smithi019.stdout:Total 225 kB/s | 509 kB 00:02 2024-03-25T19:26:22.448 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:26:22.452 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:26:22.452 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:26:22.496 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:26:22.496 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:26:22.666 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:26:22.790 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:26:23.209 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:26:23.485 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:26:23.486 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:23.486 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:26:23.486 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:23.486 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:23.486 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:26:23.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:26:24.244 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:26:36 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:26:24.402 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:26:24.402 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:24.402 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:26:24.402 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:24.402 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:26:24.402 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:26:24.402 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:24.403 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:26:24.403 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:26:24.403 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:26:24.403 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:26:24.403 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 52 M 2024-03-25T19:26:24.403 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 298 M 2024-03-25T19:26:24.403 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:27:47.292 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 637 kB/s | 52 MB 01:22 2024-03-25T19:27:47.293 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:47.293 INFO:teuthology.orchestra.run.smithi019.stdout:Total 637 kB/s | 52 MB 01:22 2024-03-25T19:27:47.293 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:27:47.295 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:27:47.295 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:27:47.646 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:27:47.646 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:27:53.901 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:27:54.016 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:27:56.072 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:27:56.373 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:27:56.374 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:27:56.374 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:27:56.374 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:56.374 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:27:56.374 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:27:56.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:27:57.126 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 1:28:09 ago on Mon 25 Mar 2024 05:59:48 PM UTC. 2024-03-25T19:27:57.281 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-25T19:27:57.282 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:27:57.282 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-25T19:27:57.282 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:27:57.282 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-25T19:27:57.282 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:27:57.282 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:27:57.283 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-25T19:27:57.283 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-25T19:27:57.283 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-25T19:27:57.283 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:27:57.283 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 1.3 M 2024-03-25T19:27:57.283 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 5.5 M 2024-03-25T19:27:57.283 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-25T19:27:59.876 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 495 kB/s | 1.3 MB 00:02 2024-03-25T19:27:59.876 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:59.877 INFO:teuthology.orchestra.run.smithi019.stdout:Total 494 kB/s | 1.3 MB 00:02 2024-03-25T19:27:59.877 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-25T19:27:59.881 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-25T19:27:59.881 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-25T19:27:59.928 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-25T19:27:59.929 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-25T19:28:00.178 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-25T19:28:00.269 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:28:00.676 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:28:00.957 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:28:00.957 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:00.957 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-25T19:28:00.957 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:28:00.957 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:00.957 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-25T19:28:01.163 DEBUG:teuthology.parallel:result is None 2024-03-25T19:28:01.164 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:28:01.310 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:28:01.331 INFO:teuthology.orchestra.run.smithi019.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:28:01.331 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:28:01.331 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:28:01.333 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:28:01.547 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:28:01.571 INFO:teuthology.orchestra.run.smithi188.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:28:01.571 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:28:01.571 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:28:01.573 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:28:01.573 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:01.573 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:28:01.600 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:01.600 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:28:01.642 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:28:01.642 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:01.642 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:28:01.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:28:01.736 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:01.736 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:28:01.767 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:28:01.836 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:28:01.836 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:01.836 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:28:01.862 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:28:01.928 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:01.928 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:28:01.957 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:28:02.026 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:28:02.026 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:02.026 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:28:02.052 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:28:02.119 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:02.119 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:28:02.147 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:28:02.215 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:28:02.298 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:28:02.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /var/log/ceph 2024-03-25T19:28:02.301 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /var/log/ceph 2024-03-25T19:28:02.328 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:28:02.329 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:28:02.368 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:28:02.395 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:28:02.395 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:28:02.434 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:28:02.472 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:28:02.472 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:28:02.473 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621310', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi019.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIi43x2E99LNSF/3BhT5R/K4DVqzzb5IcZBBtUyCOAS2Q2zD3i8ZiImheSbixMlHx8DOIpUst3eNJrQHI7tnX70=', 'smithi188.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0PwWWJMGrLoeOuDecx/Tzk2ixaznhC4GV2yeyjfgWbXxDkUGy7GYa1rGtCFqD+fTjGcat1iHyLvQ8iavV7mGw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:28:02.473 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:28:02.500 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:28:02.517 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:28:02.558 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:28:02.585 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:02.585 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:28:02.634 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:28:02.634 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2024-03-25T19:28:02.688 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:28:02.688 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:02.688 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T19:28:02.688 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:02.689 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:02.689 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-25 18:05:19.695795704 +0000 2024-03-25T19:28:02.689 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-25 17:58:33.317859761 +0000 2024-03-25T19:28:02.689 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-25 17:58:33.317859761 +0000 2024-03-25T19:28:02.689 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-25 17:58:33.317859761 +0000 2024-03-25T19:28:02.689 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:28:02.753 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-25T19:28:02.754 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-25T19:28:02.754 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000200819 s, 2.5 MB/s 2024-03-25T19:28:02.755 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:28:02.810 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2024-03-25T19:28:02.864 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:28:02.864 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:02.864 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T19:28:02.864 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:02.864 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:02.864 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-25 18:05:19.695795704 +0000 2024-03-25T19:28:02.865 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-25 17:58:33.667856229 +0000 2024-03-25T19:28:02.865 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-25 17:58:33.667856229 +0000 2024-03-25T19:28:02.865 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-25 17:58:33.667856229 +0000 2024-03-25T19:28:02.865 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:28:02.929 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-25T19:28:02.929 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-25T19:28:02.930 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000184541 s, 2.8 MB/s 2024-03-25T19:28:02.931 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:28:02.986 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2024-03-25T19:28:03.040 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:28:03.040 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:03.040 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T19:28:03.040 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:03.040 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:03.041 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-25 18:05:19.695795704 +0000 2024-03-25T19:28:03.041 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-25 17:58:34.002852847 +0000 2024-03-25T19:28:03.041 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-25 17:58:34.002852847 +0000 2024-03-25T19:28:03.041 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-25 17:58:34.002852847 +0000 2024-03-25T19:28:03.041 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:28:03.105 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-25T19:28:03.105 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-25T19:28:03.105 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000174993 s, 2.9 MB/s 2024-03-25T19:28:03.110 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:28:03.166 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2024-03-25T19:28:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:28:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:03.220 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T19:28:03.220 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:03.220 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:03.220 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-25 18:05:19.696795694 +0000 2024-03-25T19:28:03.220 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-25 17:58:34.343849405 +0000 2024-03-25T19:28:03.220 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-25 17:58:34.343849405 +0000 2024-03-25T19:28:03.220 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-25 17:58:34.343849405 +0000 2024-03-25T19:28:03.220 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:28:03.284 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-25T19:28:03.284 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-25T19:28:03.284 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000197479 s, 2.6 MB/s 2024-03-25T19:28:03.286 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:28:03.340 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:28:03.341 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:03.341 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:28:03.359 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:28:03.359 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2024-03-25T19:28:03.416 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:28:03.417 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:03.417 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T19:28:03.417 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:03.417 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:03.417 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-03-25 18:06:01.783542306 +0000 2024-03-25T19:28:03.417 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-03-25 17:58:27.022598713 +0000 2024-03-25T19:28:03.417 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-03-25 17:58:27.022598713 +0000 2024-03-25T19:28:03.417 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-03-25 17:58:27.022598713 +0000 2024-03-25T19:28:03.417 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:28:03.486 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-03-25T19:28:03.487 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-03-25T19:28:03.487 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.00023189 s, 2.2 MB/s 2024-03-25T19:28:03.488 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:28:03.547 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2024-03-25T19:28:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:28:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:03.604 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T19:28:03.604 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:03.604 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:03.604 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-03-25 18:06:01.784542286 +0000 2024-03-25T19:28:03.604 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-03-25 17:58:27.359591998 +0000 2024-03-25T19:28:03.604 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-03-25 17:58:27.359591998 +0000 2024-03-25T19:28:03.604 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-03-25 17:58:27.359591998 +0000 2024-03-25T19:28:03.604 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:28:03.673 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-03-25T19:28:03.673 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-03-25T19:28:03.673 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000207314 s, 2.5 MB/s 2024-03-25T19:28:03.675 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:28:03.732 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2024-03-25T19:28:03.789 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:28:03.789 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:03.789 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T19:28:03.789 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:03.789 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:03.789 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-03-25 18:06:01.784542286 +0000 2024-03-25T19:28:03.790 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-03-25 17:58:27.695585303 +0000 2024-03-25T19:28:03.790 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-03-25 17:58:27.695585303 +0000 2024-03-25T19:28:03.790 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-03-25 17:58:27.695585303 +0000 2024-03-25T19:28:03.790 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:28:03.858 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-03-25T19:28:03.858 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-03-25T19:28:03.858 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.00022563 s, 2.3 MB/s 2024-03-25T19:28:03.859 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:28:03.916 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2024-03-25T19:28:03.972 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:28:03.972 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:03.973 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T19:28:03.973 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:03.973 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:03.973 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-03-25 18:06:01.785542267 +0000 2024-03-25T19:28:03.973 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-03-25 17:58:28.044578348 +0000 2024-03-25T19:28:03.973 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-03-25 17:58:28.044578348 +0000 2024-03-25T19:28:03.973 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-03-25 17:58:28.044578348 +0000 2024-03-25T19:28:03.973 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:28:04.039 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-03-25T19:28:04.039 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-03-25T19:28:04.039 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000235687 s, 2.2 MB/s 2024-03-25T19:28:04.041 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:28:04.098 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:28:04.098 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi188.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:28:04.098 INFO:tasks.ceph:Generating config... 2024-03-25T19:28:04.099 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:28:04.099 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:28:04.100 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:28:04.100 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:28:04.100 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:28:04.100 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:28:04.100 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:28:04.100 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:28:04.100 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:28:04.100 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:28:04.100 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:28:04.100 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:28:04.100 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:28:04.100 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:28:04.100 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:28:04.100 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:28:04.100 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:28:04.100 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:28:04.101 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:28:04.101 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:28:04.101 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:28:04.101 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:28:04.101 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:28:04.101 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:28:04.101 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:28:04.101 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:28:04.101 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:28:04.101 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:28:04.101 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:28:04.101 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:28:04.101 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:28:04.101 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:28:04.101 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:28:04.101 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:28:04.102 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:28:04.103 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:28:04.103 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:28:04.103 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:28:04.103 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:28:04.103 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:28:04.103 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:28:04.165 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:28:04.168 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:28:04.214 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:28:04.241 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.19:6789]'), ('mon.c', '[v1:172.21.15.19:6790]'), ('mon.b', '[v1:172.21.15.188:6789]')] 2024-03-25T19:28:04.241 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.19:6789],[v1:172.21.15.19:6790],[v1:172.21.15.188:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:28:04.242 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:04.242 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:28:04.296 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.19:6789]' --addv c '[v1:172.21.15.19:6790]' --addv b '[v1:172.21.15.188:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:04.381 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:04.381 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: generated fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:04.381 INFO:teuthology.orchestra.run.smithi019.stdout:setting min_mon_release = quincy 2024-03-25T19:28:04.381 INFO:teuthology.orchestra.run.smithi019.stdout:epoch 0 2024-03-25T19:28:04.381 INFO:teuthology.orchestra.run.smithi019.stdout:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:04.382 INFO:teuthology.orchestra.run.smithi019.stdout:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:04.382 INFO:teuthology.orchestra.run.smithi019.stdout:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:04.382 INFO:teuthology.orchestra.run.smithi019.stdout:min_mon_release 17 (quincy) 2024-03-25T19:28:04.382 INFO:teuthology.orchestra.run.smithi019.stdout:election_strategy: 3 2024-03-25T19:28:04.382 INFO:teuthology.orchestra.run.smithi019.stdout:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:04.382 INFO:teuthology.orchestra.run.smithi019.stdout:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:04.382 INFO:teuthology.orchestra.run.smithi019.stdout:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:04.382 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:28:04.383 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:28:04.398 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6efd308f-8e81-46e0-802b-9ba73517b91b... 2024-03-25T19:28:04.399 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:28:04.440 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:28:04.488 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: chdir = "" 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: auth supported = cephx 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: filestore xattr use omap = true 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: mon clock drift allowed = 1.000 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: osd crush chooseleaf type = 0 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: auth debug = true 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on old message = true 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on bug = true 2024-03-25T19:28:04.489 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: mon pg warn max object skew = 0 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default size = 2 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow primary affinity = true 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow pg remap = true 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on no sortbitwise = false 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:28:04.490 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on too few osds = false 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: mon_allow_pool_size_one = true 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: osd default data pool replay window = 5 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: mon allow pool delete = true 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: mon cluster log file level = debug 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: debug asserts on shutdown = true 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: mon health detail to clog = false 2024-03-25T19:28:04.491 INFO:teuthology.orchestra.run.smithi019.stdout: mon host = "[v1:172.21.15.19:6789],[v1:172.21.15.19:6790],[v1:172.21.15.188:6789]" 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: mon client directed command retry = 5 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: mon election default strategy = 3 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: ms bind msgr2 = False 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: ms inject socket failures = 5000 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: ms type = async 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: osd heartbeat grace = 80 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout:[osd] 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: osd journal size = 100 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub load threshold = 5.0 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub max interval = 600 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.492 INFO:teuthology.orchestra.run.smithi019.stdout: osd recover clone overlap = true 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: osd recovery max chunk = 1048576 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug shutdown = true 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug op order = true 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify stray on activate = true 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug trim objects = true 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: osd open classes on start = true 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug pg log writeout = true 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: osd map max advance = 10 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: journal zero on create = true 2024-03-25T19:28:04.493 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: filestore apply finisher threads = 3 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: bdev debug aio = true 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug misdirected ops = true 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: bdev async discard = True 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: bdev enable discard = True 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore allocator = stupid 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore block size = 96636764160 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore fsck on mount = True 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore zero block detection = True 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluefs = 1 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluestore = 1 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: debug monc = 20 2024-03-25T19:28:04.494 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 20 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: debug rocksdb = 10 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd full ratio = 0.9 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: osd class default list = * 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: osd class load list = * 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: osd client watch timeout = 120 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify cached snaps = True 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify missing on start = True 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: osd fast shutdown = False 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: osd max markdown count = 1000 2024-03-25T19:28:04.495 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock override recovery settings = True 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: osd objectstore = bluestore 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: osd op queue = debug_random 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: osd op queue cut off = debug_random 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: osd shutdown pgref assert = True 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr] 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 20 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: mgr/telemetry/nag = false 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.496 INFO:teuthology.orchestra.run.smithi019.stdout:[mon] 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 20 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: mon data avail warn = 5 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: mon mgr mkfs grace = 240 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd reporter subtree level = osd 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd prime pg temp = true 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:28:04.497 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: # 1m isn't quite enough 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: mon min osdmap epochs = 50 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd crush smoke test = False 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: mon osdmap full prune interval = 2 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: mon osdmap full prune min = 15 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: mon scrub interval = 300 2024-03-25T19:28:04.498 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on pool no app = False 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout: paxos service trim min = 10 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout:[client] 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout: rgw cache enabled = true 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable ops log = true 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable usage log = true 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout: debug monc = 20 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout: debug objecter = 20 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout: debug rados = 20 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.a] 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.c] 2024-03-25T19:28:04.499 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.b] 2024-03-25T19:28:04.500 INFO:teuthology.orchestra.run.smithi188.stdout:[global] 2024-03-25T19:28:04.500 INFO:teuthology.orchestra.run.smithi188.stdout: chdir = "" 2024-03-25T19:28:04.500 INFO:teuthology.orchestra.run.smithi188.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:28:04.500 INFO:teuthology.orchestra.run.smithi188.stdout: auth supported = cephx 2024-03-25T19:28:04.500 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.500 INFO:teuthology.orchestra.run.smithi188.stdout: filestore xattr use omap = true 2024-03-25T19:28:04.500 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.500 INFO:teuthology.orchestra.run.smithi188.stdout: mon clock drift allowed = 1.000 2024-03-25T19:28:04.500 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: osd crush chooseleaf type = 0 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: auth debug = true 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on old message = true 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on bug = true 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: mon pg warn max object skew = 0 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default size = 2 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.501 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow primary affinity = true 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow pg remap = true 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on no sortbitwise = false 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on too few osds = false 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: mon_allow_pool_size_one = true 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: osd default data pool replay window = 5 2024-03-25T19:28:04.502 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: mon allow pool delete = true 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: mon cluster log file level = debug 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: debug asserts on shutdown = true 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: mon health detail to clog = false 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: mon host = "[v1:172.21.15.19:6789],[v1:172.21.15.19:6790],[v1:172.21.15.188:6789]" 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: mon client directed command retry = 5 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: mon election default strategy = 3 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: ms bind msgr2 = False 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: ms inject socket failures = 5000 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: ms type = async 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: osd heartbeat grace = 80 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: fsid = 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.503 INFO:teuthology.orchestra.run.smithi188.stdout:[osd] 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: osd journal size = 100 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub load threshold = 5.0 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub max interval = 600 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: osd recover clone overlap = true 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: osd recovery max chunk = 1048576 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug shutdown = true 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug op order = true 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify stray on activate = true 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug trim objects = true 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.504 INFO:teuthology.orchestra.run.smithi188.stdout: osd open classes on start = true 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug pg log writeout = true 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: osd map max advance = 10 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: journal zero on create = true 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: filestore apply finisher threads = 3 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: bdev debug aio = true 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug misdirected ops = true 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: bdev async discard = True 2024-03-25T19:28:04.505 INFO:teuthology.orchestra.run.smithi188.stdout: bdev enable discard = True 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore allocator = stupid 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore block size = 96636764160 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore fsck on mount = True 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore zero block detection = True 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: debug bluefs = 1 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: debug bluestore = 1 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: debug monc = 20 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: debug osd = 20 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: debug rocksdb = 10 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd full ratio = 0.9 2024-03-25T19:28:04.506 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd class default list = * 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd class load list = * 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd client watch timeout = 120 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify cached snaps = True 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify missing on start = True 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd fast shutdown = False 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd max markdown count = 1000 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd mclock override recovery settings = True 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd objectstore = bluestore 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd op queue = debug_random 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd op queue cut off = debug_random 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: osd shutdown pgref assert = True 2024-03-25T19:28:04.507 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr] 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: debug mgr = 20 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: mgr/telemetry/nag = false 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout:[mon] 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: debug paxos = 20 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2024-03-25T19:28:04.508 INFO:teuthology.orchestra.run.smithi188.stdout: mon data avail warn = 5 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: mon mgr mkfs grace = 240 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd reporter subtree level = osd 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd prime pg temp = true 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.509 INFO:teuthology.orchestra.run.smithi188.stdout: # 1m isn't quite enough 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: mon min osdmap epochs = 50 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd crush smoke test = False 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: mon osdmap full prune interval = 2 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: mon osdmap full prune min = 15 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: mon scrub interval = 300 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on pool no app = False 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: paxos service trim min = 10 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout:[client] 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: rgw cache enabled = true 2024-03-25T19:28:04.510 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable ops log = true 2024-03-25T19:28:04.511 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable usage log = true 2024-03-25T19:28:04.511 INFO:teuthology.orchestra.run.smithi188.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:28:04.511 INFO:teuthology.orchestra.run.smithi188.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:28:04.511 INFO:teuthology.orchestra.run.smithi188.stdout: debug monc = 20 2024-03-25T19:28:04.511 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-03-25T19:28:04.511 INFO:teuthology.orchestra.run.smithi188.stdout: debug objecter = 20 2024-03-25T19:28:04.511 INFO:teuthology.orchestra.run.smithi188.stdout: debug rados = 20 2024-03-25T19:28:04.511 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.a] 2024-03-25T19:28:04.511 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.c] 2024-03-25T19:28:04.511 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.b] 2024-03-25T19:28:04.512 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:28:04.513 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:28:04.568 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:28:04.568 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:04.568 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:28:04.582 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:04.582 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:28:04.637 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi019.front.sepia.ceph.com 2024-03-25T19:28:04.637 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:04.637 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:28:04.637 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:28:04.715 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:04.715 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:04.770 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi188.front.sepia.ceph.com 2024-03-25T19:28:04.770 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:04.770 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:28:04.770 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:28:04.811 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:04.811 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:04.867 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:28:04.867 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:28:04.867 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:28:04.921 INFO:teuthology.orchestra.run.smithi019.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:28:04.923 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:28:04.989 INFO:teuthology.orchestra.run.smithi188.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:28:04.992 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:28:04.992 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:28:04.993 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:28:05.034 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:28:05.049 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:28:05.092 INFO:teuthology.orchestra.run.smithi188.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:28:05.107 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:28:05.107 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi188.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:28:05.107 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:28:05.133 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:28:05.133 INFO:tasks.ceph:role: osd.0 2024-03-25T19:28:05.133 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-25T19:28:05.133 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:28:05.207 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:28:05.207 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:28:05.208 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:28:05.208 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:28:05.208 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:28:05.208 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:28:05.208 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:28:05.208 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:28:05.208 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:28:05.208 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:28:05.715 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-03-25T19:28:05.717 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-03-25T19:28:05.717 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:28:05.837 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:28:05.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:28:05.939 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:28:05.939 INFO:tasks.ceph:role: osd.1 2024-03-25T19:28:05.939 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-25T19:28:05.939 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:28:06.013 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:28:06.013 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:28:06.013 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:28:06.013 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:28:06.013 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:28:06.013 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:28:06.013 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:28:06.013 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:28:06.013 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:28:06.013 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:28:06.472 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-03-25T19:28:06.474 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-03-25T19:28:06.474 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:28:06.567 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:28:06.597 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:28:06.664 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:28:06.664 INFO:tasks.ceph:role: osd.2 2024-03-25T19:28:06.665 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-25T19:28:06.665 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:28:06.738 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:28:06.739 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:28:06.739 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:28:06.739 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:28:06.739 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:28:06.739 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:28:06.739 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:28:06.739 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:28:06.739 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:28:06.739 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:28:07.235 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-03-25T19:28:07.237 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-03-25T19:28:07.238 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:28:07.387 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:28:07.418 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:28:07.484 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:28:07.484 INFO:tasks.ceph:role: osd.3 2024-03-25T19:28:07.484 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-25T19:28:07.484 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:28:07.558 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:28:07.558 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:28:07.558 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:28:07.558 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:28:07.558 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:28:07.558 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:28:07.559 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:28:07.559 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:28:07.559 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:28:07.559 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:28:08.072 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-03-25T19:28:08.074 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-03-25T19:28:08.075 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:28:08.224 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:28:08.258 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:08.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:08.365+0000 7fbeabc37640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:28:08.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:08.366+0000 7fbeabc37640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:28:08.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:08.366+0000 7fbeabc37640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:28:08.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:08.366+0000 7fbeabc37640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:28:08.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:08.366+0000 7fbeabc37640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:28:08.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:08.366+0000 7fbeabc37640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:28:11.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:28:11.242 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:11.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:11.332+0000 7fc4d78f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:28:11.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:11.332+0000 7fc4d78f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:28:11.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:11.332+0000 7fc4d78f9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:28:11.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:11.333+0000 7fc4d78f9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:28:11.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:11.333+0000 7fc4d78f9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:28:11.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:11.333+0000 7fc4d78f9640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:28:14.175 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:28:14.206 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:14.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:14.297+0000 7fe386208640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:28:14.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:14.297+0000 7fe386208640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:28:14.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:14.297+0000 7fe386208640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:28:14.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:14.297+0000 7fe386208640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:28:14.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:14.297+0000 7fe386208640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:28:14.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:14.298+0000 7fe386208640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:28:17.141 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:28:17.172 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:17.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:17.264+0000 7f5d6dc11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:28:17.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:17.265+0000 7f5d6dc11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:28:17.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:17.265+0000 7f5d6dc11640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:28:17.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:17.265+0000 7f5d6dc11640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:28:17.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:17.265+0000 7f5d6dc11640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:28:17.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:17.265+0000 7f5d6dc11640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:28:20.105 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:28:20.136 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:28:20.168 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:28:20.168 INFO:tasks.ceph:role: osd.4 2024-03-25T19:28:20.168 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi188.front.sepia.ceph.com 2024-03-25T19:28:20.168 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:28:20.246 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:28:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:28:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:28:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:28:20.246 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:28:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:28:20.246 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:28:20.246 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:28:20.246 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:28:20.246 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:28:20.754 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-03-25T19:28:20.757 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-03-25T19:28:20.757 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:28:20.909 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:28:20.947 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:28:21.016 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:28:21.017 INFO:tasks.ceph:role: osd.5 2024-03-25T19:28:21.017 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi188.front.sepia.ceph.com 2024-03-25T19:28:21.017 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:28:21.092 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:28:21.092 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:28:21.092 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:28:21.092 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:28:21.092 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:28:21.093 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:28:21.093 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:28:21.093 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:28:21.093 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:28:21.093 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:28:21.528 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-03-25T19:28:21.530 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-03-25T19:28:21.530 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:28:21.627 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:28:21.659 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:28:21.727 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:28:21.727 INFO:tasks.ceph:role: osd.6 2024-03-25T19:28:21.727 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi188.front.sepia.ceph.com 2024-03-25T19:28:21.727 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:28:21.802 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:28:21.802 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:28:21.802 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:28:21.802 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:28:21.802 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:28:21.802 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:28:21.803 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:28:21.803 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:28:21.803 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:28:21.803 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:28:22.300 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-03-25T19:28:22.302 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-03-25T19:28:22.303 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:28:22.423 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:28:22.457 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:28:22.523 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:28:22.523 INFO:tasks.ceph:role: osd.7 2024-03-25T19:28:22.523 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi188.front.sepia.ceph.com 2024-03-25T19:28:22.523 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:28:22.598 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:28:22.598 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:28:22.598 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:28:22.598 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:28:22.598 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:28:22.598 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:28:22.598 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:28:22.598 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:28:22.598 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:28:22.598 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:28:23.094 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-03-25T19:28:23.096 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-03-25T19:28:23.096 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:28:23.234 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:28:23.270 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:23.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:23.372+0000 7fbb5de48640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:28:23.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:23.372+0000 7fbb5de48640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:28:23.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:23.372+0000 7fbb5de48640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:28:23.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:23.372+0000 7fbb5de48640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:28:23.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:23.372+0000 7fbb5de48640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:28:23.373 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:23.373+0000 7fbb5de48640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:28:26.217 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:28:26.248 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:26.342 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:26.341+0000 7fce478bf640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:28:26.342 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:26.342+0000 7fce478bf640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:28:26.342 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:26.342+0000 7fce478bf640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:28:26.342 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:26.342+0000 7fce478bf640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:28:26.342 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:26.342+0000 7fce478bf640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:28:26.342 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:26.342+0000 7fce478bf640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:28:29.184 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:28:29.216 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:29.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:29.309+0000 7fa4dbaae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:28:29.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:29.309+0000 7fa4dbaae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:28:29.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:29.309+0000 7fa4dbaae640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:28:29.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:29.309+0000 7fa4dbaae640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:28:29.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:29.309+0000 7fa4dbaae640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:28:29.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:29.309+0000 7fa4dbaae640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:28:32.158 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:28:32.190 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:32.283 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:32.283+0000 7fb180973640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:28:32.284 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:32.283+0000 7fb180973640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:28:32.284 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:32.283+0000 7fb180973640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:28:32.284 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:32.283+0000 7fb180973640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:28:32.284 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:32.283+0000 7fb180973640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:28:32.284 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:32.284+0000 7fb180973640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:28:35.131 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:28:35.162 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:28:35.162 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:35.163 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:28:35.195 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:35.195 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:28:35.267 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:35.267 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:28:35.336 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:35.336 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:28:35.405 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:35.405 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:28:35.475 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:35.475 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:28:35.506 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:35.506 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:28:35.578 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:35.578 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:28:35.647 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:35.647 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:28:35.716 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:35.716 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:28:35.784 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:35.784 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:28:35.801 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:35.802 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:28:35.840 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:28:35.840 DEBUG:teuthology.orchestra.run.smithi019:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:28:35.844 DEBUG:teuthology.orchestra.run.smithi188:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:28:35.883 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.y] 2024-03-25T19:28:35.884 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBE0AFmMtYBNxAA4X+34XW5csnq9q7EwiO3KA== 2024-03-25T19:28:35.884 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.0] 2024-03-25T19:28:35.884 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBI0AFmibTiFRAAMQapZS+OWj2hW+SKRrEzXQ== 2024-03-25T19:28:35.884 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.1] 2024-03-25T19:28:35.884 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBL0AFmzezmExAAmbplv6qS18xfBtZhobdOBQ== 2024-03-25T19:28:35.884 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.2] 2024-03-25T19:28:35.884 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBO0AFmMGPQERAA7YtkNbeYWhCcdu/Sq5k4JA== 2024-03-25T19:28:35.884 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.3] 2024-03-25T19:28:35.884 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBR0AFmt43bDxAAWD1Q1m0Ma3y/7IazpWpf7A== 2024-03-25T19:28:35.884 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.x] 2024-03-25T19:28:35.884 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBE0AFmQ4oSOxAAu/nvyncASqBTZXCui1BASg== 2024-03-25T19:28:35.885 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.4] 2024-03-25T19:28:35.885 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBX0AFm8to8FhAABBJwtkVzJvkV+ovfIazkYg== 2024-03-25T19:28:35.885 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.5] 2024-03-25T19:28:35.885 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBa0AFmfI1tFBAAkTHP8HJ7e0Spg1/zju3Ykw== 2024-03-25T19:28:35.885 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.6] 2024-03-25T19:28:35.885 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBd0AFmVS15EhAAvlH1EMkKQaTcOzwVrt5lDA== 2024-03-25T19:28:35.885 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.7] 2024-03-25T19:28:35.885 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBg0AFmsI3vEBAAVDe6PHyCjKsu9g4TkT/sFA== 2024-03-25T19:28:35.885 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2024-03-25T19:28:35.885 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBF0AFmg6gnAhAANWRbqWnM/rrm5ModsP/UJg== 2024-03-25T19:28:35.885 INFO:teuthology.orchestra.run.smithi019.stdout:[client.1] 2024-03-25T19:28:35.885 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQBF0AFmmrKPBRAAgh71bxJ+AYvRY3igTt7bcg== 2024-03-25T19:28:35.907 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr.y] 2024-03-25T19:28:35.907 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBE0AFmMtYBNxAA4X+34XW5csnq9q7EwiO3KA== 2024-03-25T19:28:35.907 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.0] 2024-03-25T19:28:35.907 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBI0AFmibTiFRAAMQapZS+OWj2hW+SKRrEzXQ== 2024-03-25T19:28:35.907 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.1] 2024-03-25T19:28:35.907 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBL0AFmzezmExAAmbplv6qS18xfBtZhobdOBQ== 2024-03-25T19:28:35.907 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.2] 2024-03-25T19:28:35.907 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBO0AFmMGPQERAA7YtkNbeYWhCcdu/Sq5k4JA== 2024-03-25T19:28:35.907 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.3] 2024-03-25T19:28:35.907 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBR0AFmt43bDxAAWD1Q1m0Ma3y/7IazpWpf7A== 2024-03-25T19:28:35.907 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr.x] 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBE0AFmQ4oSOxAAu/nvyncASqBTZXCui1BASg== 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.4] 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBX0AFm8to8FhAABBJwtkVzJvkV+ovfIazkYg== 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.5] 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBa0AFmfI1tFBAAkTHP8HJ7e0Spg1/zju3Ykw== 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.6] 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBd0AFmVS15EhAAvlH1EMkKQaTcOzwVrt5lDA== 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.7] 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBg0AFmsI3vEBAAVDe6PHyCjKsu9g4TkT/sFA== 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout:[client.0] 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBF0AFmg6gnAhAANWRbqWnM/rrm5ModsP/UJg== 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout:[client.1] 2024-03-25T19:28:35.908 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBF0AFmmrKPBRAAgh71bxJ+AYvRY3igTt7bcg== 2024-03-25T19:28:35.909 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:28:35.929 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:28:36.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.005 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.058 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.060 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.111 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.113 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.165 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.167 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.219 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:28:36.221 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:28:36.272 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.275 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.327 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.329 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.383 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.437 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:28:36.489 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:28:36.491 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:28:36.545 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:28:36.547 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:28:36.602 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:28:36.602 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:28:36.628 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:28:36.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:28:36.750 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:28:36.817 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:28:36.913 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:28:36.940 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:28:36.968 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:28:37.065 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:28:37.091 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:37.094 DEBUG:teuthology.orchestra.run.smithi188:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:28:37.146 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:28:37.147 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:28:37.147 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:28:37.147 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:28:37.149 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:28:37.149 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:28:37.150 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:28:37.150 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:28:37.152 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:28:37.152 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:28:37.152 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:28:37.152 DEBUG:teuthology.orchestra.run.smithi188:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:28:37.189 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:28:37.189 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:28:37.189 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:28:37.190 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:28:37.192 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:28:37.192 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:28:37.192 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:28:37.194 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:28:37.194 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:28:37.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:28:37.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.296+0000 7f8ea149a640 10 monclient: get_monmap_and_config 2024-03-25T19:28:37.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.296+0000 7f8ea149a640 10 monclient: build_initial_monmap 2024-03-25T19:28:37.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.296+0000 7f8ea149a640 10 monclient: monmap: 2024-03-25T19:28:37.297 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-03-25T19:28:37.297 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:37.297 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-25T19:28:37.297940+0000 2024-03-25T19:28:37.297 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-25T19:28:37.297940+0000 2024-03-25T19:28:37.297 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:37.297 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 3 2024-03-25T19:28:37.297 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:37.297 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:28:37.297 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:28:37.297 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:28:37.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 1 Processor -- start 2024-03-25T19:28:37.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 1 -- start start 2024-03-25T19:28:37.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 10 monclient: init 2024-03-25T19:28:37.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:37.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:37.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 10 monclient(hunting): picked mon.noname-c con 0x7f8e9c0b4700 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:37.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 10 monclient(hunting): picked mon.noname-a con 0x7f8e9c142a20 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:37.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 10 monclient(hunting): picked mon.noname-b con 0x7f8e9c0a96b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:37.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0a9d60 con 0x7f8e9c142a20 2024-03-25T19:28:37.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c13ff10 con 0x7f8e9c0b4700 2024-03-25T19:28:37.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0aa1f0 con 0x7f8e9c0a96b0 2024-03-25T19:28:37.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:37.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8ea149a640 10 monclient(hunting): authenticate will time out at 5970.567861s 2024-03-25T19:28:37.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8e927fc640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e9c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-03-25T19:28:37.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.297+0000 7f8e9b7fe640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e9c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6790/0 2024-03-25T19:28:37.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.298+0000 7f8e9affd640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e9c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.188:6789/0 2024-03-25T19:28:37.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.499+0000 7f8e9b7fe640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e9c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6790/0 2024-03-25T19:28:37.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.499+0000 7f8e927fc640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e9c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-03-25T19:28:37.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.499+0000 7f8e9affd640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e9c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.188:6789/0 2024-03-25T19:28:37.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.900+0000 7f8e9b7fe640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e9c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6790/0 2024-03-25T19:28:37.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.900+0000 7f8e927fc640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e9c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-03-25T19:28:37.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:37.900+0000 7f8e9affd640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e9c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.188:6789/0 2024-03-25T19:28:38.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:38.701+0000 7f8e9b7fe640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e9c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6790/0 2024-03-25T19:28:38.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:38.701+0000 7f8e927fc640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e9c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-03-25T19:28:38.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:38.701+0000 7f8e9affd640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e9c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.188:6789/0 2024-03-25T19:28:40.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.297+0000 7f8e997fa640 10 monclient(hunting): tick 2024-03-25T19:28:40.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.297+0000 7f8e997fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:40.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.297+0000 7f8e997fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:40.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.297+0000 7f8e997fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:40.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.297+0000 7f8e997fa640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e9c13f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:40.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e997fa640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e9c142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:40.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e997fa640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e9c13f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:40.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e997fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:40.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-b con 0x7f8e9c0a96b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:40.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-a con 0x7f8e9c142a20 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:40.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-c con 0x7f8e9c0b4700 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:40.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e997fa640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0b4ae0 con 0x7f8e9c142a20 2024-03-25T19:28:40.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e997fa640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0a9d60 con 0x7f8e9c0b4700 2024-03-25T19:28:40.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e9affd640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6790/0 2024-03-25T19:28:40.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e997fa640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c13ff10 con 0x7f8e9c0a96b0 2024-03-25T19:28:40.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e997fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:40.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e927fc640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-03-25T19:28:40.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.298+0000 7f8e9b7fe640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e7c00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.188:6789/0 2024-03-25T19:28:40.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.499+0000 7f8e9affd640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6790/0 2024-03-25T19:28:40.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.499+0000 7f8e927fc640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-03-25T19:28:40.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.499+0000 7f8e9b7fe640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e7c00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.188:6789/0 2024-03-25T19:28:40.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.900+0000 7f8e927fc640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-03-25T19:28:40.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.900+0000 7f8e9affd640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6790/0 2024-03-25T19:28:40.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:40.900+0000 7f8e9b7fe640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e7c00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.188:6789/0 2024-03-25T19:28:41.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:41.701+0000 7f8e927fc640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-03-25T19:28:41.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:41.701+0000 7f8e9affd640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6790/0 2024-03-25T19:28:41.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:41.701+0000 7f8e9b7fe640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e7c00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.188:6789/0 2024-03-25T19:28:43.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 10 monclient(hunting): tick 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c0056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e7c00c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-c con 0x7f8e9c0b4700 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-a con 0x7f8e9c142a20 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-b con 0x7f8e9c0a96b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0aa1f0 con 0x7f8e9c142a20 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0b4ae0 con 0x7f8e9c0b4700 2024-03-25T19:28:43.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0a9d60 con 0x7f8e9c0a96b0 2024-03-25T19:28:43.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e997fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:43.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e927fc640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-03-25T19:28:43.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e9b7fe640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6790/0 2024-03-25T19:28:43.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.299+0000 7f8e9affd640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e7c00cec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.188:6789/0 2024-03-25T19:28:43.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.500+0000 7f8e927fc640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-03-25T19:28:43.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.500+0000 7f8e9b7fe640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6790/0 2024-03-25T19:28:43.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.500+0000 7f8e9affd640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e7c00cec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.188:6789/0 2024-03-25T19:28:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.901+0000 7f8e927fc640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-03-25T19:28:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.901+0000 7f8e9b7fe640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6790/0 2024-03-25T19:28:43.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:43.901+0000 7f8e9affd640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e7c00cec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.188:6789/0 2024-03-25T19:28:44.172 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:00:06.875 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:44.480 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:00:07.183 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:44.678 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:00:07.408 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:44.681 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:00:07.411 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:44.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:44.702+0000 7f8e9b7fe640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6790/0 2024-03-25T19:28:44.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:44.702+0000 7f8e927fc640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c00c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-03-25T19:28:44.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:44.702+0000 7f8e9affd640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e7c00cec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.188:6789/0 2024-03-25T19:28:45.006 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:00:07.735 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:45.007 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:00:07.736 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:46.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 10 monclient(hunting): tick 2024-03-25T19:28:46.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:46.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:46.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:46.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 1 -- >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c0056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:46.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 1 -- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e7c00cec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:46.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c00c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:28:46.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:46.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-a con 0x7f8e9c142a20 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:46.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-c con 0x7f8e9c0b4700 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:46.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-b con 0x7f8e9c0a96b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:46.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c13ff10 con 0x7f8e9c142a20 2024-03-25T19:28:46.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0aa1f0 con 0x7f8e9c0b4700 2024-03-25T19:28:46.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0b4ae0 con 0x7f8e9c0a96b0 2024-03-25T19:28:46.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.299+0000 7f8e997fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:46.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.355+0000 7f8e927fc640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 0x7f8e7c00c770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:51694/0 (socket says 172.21.15.19:51694) 2024-03-25T19:28:46.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.355+0000 7f8e927fc640 1 -- 172.21.15.19:0/3958386128 learned_addr learned my addr 172.21.15.19:0/3958386128 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:28:46.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.678+0000 7f8e9b7fe640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c00cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T19:28:46.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.678+0000 7f8e9b7fe640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c00cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:28:46.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.678+0000 7f8e9b7fe640 1 --1- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 0x7f8e7c00cec0 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-03-25T19:28:46.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.678+0000 7f8e9b7fe640 1 --1- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 0x7f8e7c00cec0 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:28:46.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.678+0000 7f8e9a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.19:6790/0 2024-03-25T19:28:46.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.699+0000 7f8e9affd640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c008e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T19:28:46.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.699+0000 7f8e9affd640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c008e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:28:46.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:46.699+0000 7f8e9affd640 1 --1- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 0x7f8e7c008e90 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:28:49.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 10 monclient(hunting): tick 2024-03-25T19:28:49.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:49.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:49.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:49.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c00cec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:28:49.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.188:6789/0 conn(0x7f8e9c0b4700 legacy=0x7f8e7c00c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:49.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6789/0 conn(0x7f8e9c142a20 legacy=0x7f8e7c008e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:49.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:49.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-c con 0x7f8e7c005740 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:49.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-a con 0x7f8e7c0120c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:49.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-b con 0x7f8e9c0a96b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:49.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0a9d60 con 0x7f8e7c0120c0 2024-03-25T19:28:49.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c008e70 con 0x7f8e7c005740 2024-03-25T19:28:49.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c009000 con 0x7f8e9c0a96b0 2024-03-25T19:28:49.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:49.299+0000 7f8e997fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:52.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 10 monclient(hunting): tick 2024-03-25T19:28:52.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:52.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:52.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:52.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6790/0 conn(0x7f8e9c0a96b0 legacy=0x7f8e7c00cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:52.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.188:6789/0 conn(0x7f8e7c005740 legacy=0x7f8e7c00d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:52.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6789/0 conn(0x7f8e7c0120c0 legacy=0x7f8e7c0144b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:52.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:52.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-a con 0x7f8e7c0095d0 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:52.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-b con 0x7f8e7c00a120 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:52.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-c con 0x7f8e7c016be0 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:52.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c009190 con 0x7f8e7c0095d0 2024-03-25T19:28:52.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c00d710 con 0x7f8e7c016be0 2024-03-25T19:28:52.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c00d940 con 0x7f8e7c00a120 2024-03-25T19:28:52.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:52.300+0000 7f8e997fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:55.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 10 monclient(hunting): tick 2024-03-25T19:28:55.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:28:55.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:28:55.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:28:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6790/0 conn(0x7f8e7c00a120 legacy=0x7f8e7c00c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.188:6789/0 conn(0x7f8e7c016be0 legacy=0x7f8e7c003680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6789/0 conn(0x7f8e7c0095d0 legacy=0x7f8e7c009a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-a con 0x7f8e7c00cec0 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-b con 0x7f8e7c00a120 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 10 monclient(hunting): picked mon.noname-c con 0x7f8e7c016be0 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c00db70 con 0x7f8e7c00cec0 2024-03-25T19:28:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c00d2f0 con 0x7f8e7c016be0 2024-03-25T19:28:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7c0099b0 con 0x7f8e7c00a120 2024-03-25T19:28:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.300+0000 7f8e997fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:55.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.887+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254363300 0 0) 0x7f8e7c00d2f0 con 0x7f8e7c016be0 2024-03-25T19:28:55.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.887+0000 7f8e9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:55.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.887+0000 7f8e9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:55.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.887+0000 7f8e9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:55.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.887+0000 7f8e9a7fc640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:28:55.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.887+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e88003520 con 0x7f8e7c016be0 2024-03-25T19:28:55.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.897+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097739793 0 0) 0x7f8e7c00db70 con 0x7f8e7c00cec0 2024-03-25T19:28:55.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.897+0000 7f8e9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:55.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.897+0000 7f8e9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:55.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.897+0000 7f8e9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:55.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.897+0000 7f8e9a7fc640 10 monclient(hunting): my global_id is 4104 2024-03-25T19:28:55.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.898+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e7c00d2f0 con 0x7f8e7c00cec0 2024-03-25T19:28:55.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.936+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149340435 0 0) 0x7f8e7c0099b0 con 0x7f8e7c00a120 2024-03-25T19:28:55.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.936+0000 7f8e9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:55.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.936+0000 7f8e9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:55.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.936+0000 7f8e9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:55.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.936+0000 7f8e9a7fc640 10 monclient(hunting): my global_id is 4103 2024-03-25T19:28:55.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.936+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e7c00db70 con 0x7f8e7c00a120 2024-03-25T19:28:55.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4159573742 0 0) 0x7f8e7c00d2f0 con 0x7f8e7c00cec0 2024-03-25T19:28:55.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6790/0 conn(0x7f8e7c00a120 legacy=0x7f8e7c00c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.188:6789/0 conn(0x7f8e7c016be0 legacy=0x7f8e7c003660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:55.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:28:55.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c009b90 con 0x7f8e7c00cec0 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8e8c0040c0 con 0x7f8e7c00cec0 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8ea149a640 5 monclient: authenticate success, global_id 4104 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8ea149a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8e9a7fc640 10 monclient: dump: 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:55.944 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:55.945 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:55.945 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 3 2024-03-25T19:28:55.945 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:55.945 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:55.945 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:55.945 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:28:55.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.944+0000 7f8ea149a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:55.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.995+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e8c004220 con 0x7f8e7c00cec0 2024-03-25T19:28:55.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.995+0000 7f8e9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:55.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.995+0000 7f8e9a7fc640 1 -- 172.21.15.19:0/3958386128 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8e8c004630 con 0x7f8e7c00cec0 2024-03-25T19:28:55.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.995+0000 7f8e9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:55.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.995+0000 7f8e9a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:55.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.995+0000 7f8e9a7fc640 10 monclient: dump: 2024-03-25T19:28:55.995 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-25T19:28:55.995 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:55.995 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:55.995 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:55.995 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:55.995 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 3 2024-03-25T19:28:55.995 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:55.995 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:55.995 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:55.995 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:28:55.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.995+0000 7f8ea149a640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:55.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.995+0000 7f8ea149a640 4 set_mon_vals no callback set 2024-03-25T19:28:55.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.996+0000 7f8ea149a640 10 monclient: shutdown 2024-03-25T19:28:55.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.996+0000 7f8ea149a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:55.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.996+0000 7f8ea149a640 1 -- 172.21.15.19:0/3958386128 >> v1:172.21.15.19:6789/0 conn(0x7f8e7c00cec0 legacy=0x7f8e7c005e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.996+0000 7f8ea149a640 1 -- 172.21.15.19:0/3958386128 shutdown_connections 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.996+0000 7f8ea149a640 1 -- 172.21.15.19:0/3958386128 >> 172.21.15.19:0/3958386128 conn(0x7f8e9c0668e0 msgr2=0x7f8e9c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.996+0000 7f8ea149a640 1 -- 172.21.15.19:0/3958386128 shutdown_connections 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.996+0000 7f8ea149a640 1 -- 172.21.15.19:0/3958386128 wait complete. 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.997+0000 7f8ea149a640 10 monclient: build_initial_monmap 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.997+0000 7f8ea149a640 10 monclient: monmap: 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:55.998 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.997+0000 7f8ea149a640 1 librados: starting msgr at 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.997+0000 7f8ea149a640 1 librados: starting objecter 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.997+0000 7f8ea149a640 1 Processor -- start 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.997+0000 7f8ea149a640 1 -- start start 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.997+0000 7f8ea149a640 1 librados: setting wanted keys 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.997+0000 7f8ea149a640 1 librados: calling monclient init 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.997+0000 7f8ea149a640 10 monclient: init 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.998+0000 7f8ea149a640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.998+0000 7f8ea149a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.998+0000 7f8ea149a640 10 monclient(hunting): picked mon.noname-c con 0x7f8e7c00a120 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.998+0000 7f8ea149a640 10 monclient(hunting): picked mon.noname-b con 0x7f8e7c00cec0 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:55.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.998+0000 7f8ea149a640 10 monclient(hunting): picked mon.noname-a con 0x7f8e7c016be0 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:56.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.998+0000 7f8ea149a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0b0a60 con 0x7f8e7c016be0 2024-03-25T19:28:56.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.998+0000 7f8ea149a640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c0b1160 con 0x7f8e7c00cec0 2024-03-25T19:28:56.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.998+0000 7f8ea149a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9c138c60 con 0x7f8e7c00a120 2024-03-25T19:28:56.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.998+0000 7f8ea149a640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:56.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:55.998+0000 7f8ea149a640 10 monclient(hunting): authenticate will time out at 5989.267657s 2024-03-25T19:28:56.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.000+0000 7f8e927fc640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f8e7c00cec0 0x7f8e9c0af030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:55994/0 (socket says 172.21.15.19:55994) 2024-03-25T19:28:56.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.000+0000 7f8e927fc640 1 -- 172.21.15.19:0/2905489702 learned_addr learned my addr 172.21.15.19:0/2905489702 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:28:56.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.002+0000 7f8e93fff640 10 client.?.objecter ms_handle_connect 0x7f8e7c00cec0 2024-03-25T19:28:56.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.002+0000 7f8e93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:56.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.003+0000 7f8e93fff640 10 client.?.objecter ms_handle_connect 0x7f8e7c016be0 2024-03-25T19:28:56.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.003+0000 7f8e93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:56.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.008+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590482617 0 0) 0x7f8e9c0b0a60 con 0x7f8e7c016be0 2024-03-25T19:28:56.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.008+0000 7f8e93fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:56.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.008+0000 7f8e93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:56.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.008+0000 7f8e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:56.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.008+0000 7f8e93fff640 10 monclient(hunting): my global_id is 4113 2024-03-25T19:28:56.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.008+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e7c0099b0 con 0x7f8e7c016be0 2024-03-25T19:28:56.029 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.030+0000 7f8e93fff640 10 client.?.objecter ms_handle_connect 0x7f8e7c00a120 2024-03-25T19:28:56.029 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.030+0000 7f8e93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:56.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.053+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2615200606 0 0) 0x7f8e7c0099b0 con 0x7f8e7c016be0 2024-03-25T19:28:56.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.053+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e9c0b0a60 con 0x7f8e7c016be0 2024-03-25T19:28:56.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.053+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8e94004540 con 0x7f8e7c016be0 2024-03-25T19:28:56.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.053+0000 7f8e93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:56.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.053+0000 7f8e93fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:56.052 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.053+0000 7f8e93fff640 10 monclient(hunting): dump: 2024-03-25T19:28:56.052 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-25T19:28:56.052 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:56.052 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:56.052 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:56.052 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:56.052 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 3 2024-03-25T19:28:56.053 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:56.053 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:56.053 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:56.053 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:28:56.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.053+0000 7f8e93fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:56.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1927176732 0 0) 0x7f8e9c0b0a60 con 0x7f8e7c016be0 2024-03-25T19:28:56.105 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 >> v1:172.21.15.19:6790/0 conn(0x7f8e7c00a120 legacy=0x7f8e9c1d7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:56.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 >> v1:172.21.15.188:6789/0 conn(0x7f8e7c00cec0 legacy=0x7f8e9c0af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:56.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8e93fff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:56.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8e93fff640 1 monclient: found mon.a 2024-03-25T19:28:56.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8e93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:56.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8e93fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:56.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c0b16b0 con 0x7f8e7c016be0 2024-03-25T19:28:56.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8e93fff640 10 monclient: _finish_auth 0 2024-03-25T19:28:56.106 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8e93fff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:56.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8e93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:56.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8ea149a640 5 monclient: authenticate success, global_id 4113 2024-03-25T19:28:56.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8ea149a640 10 monclient: _renew_subs 2024-03-25T19:28:56.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8ea149a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:56.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8ea149a640 1 -- 172.21.15.19:0/2905489702 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9c1d8300 con 0x7f8e7c016be0 2024-03-25T19:28:56.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8ea149a640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:56.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8ea149a640 10 monclient: _renew_subs 2024-03-25T19:28:56.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8ea149a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:56.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8ea149a640 1 -- 172.21.15.19:0/2905489702 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e9c0af920 con 0x7f8e7c016be0 2024-03-25T19:28:56.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.106+0000 7f8ea149a640 1 librados: init done 2024-03-25T19:28:56.108 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.107+0000 7f8ea149a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:56.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.107+0000 7f8ea149a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:56.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.107+0000 7f8ea149a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:56.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.107+0000 7f8ea149a640 1 -- 172.21.15.19:0/2905489702 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9c0b3a70 con 0x7f8e7c016be0 2024-03-25T19:28:56.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e94004950 con 0x7f8e7c016be0 2024-03-25T19:28:56.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:56.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8e94004d20 con 0x7f8e7c016be0 2024-03-25T19:28:56.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:56.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:56.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 10 monclient: dump: 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 3 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8e940052d0 con 0x7f8e7c016be0 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e99ffb640 4 set_mon_vals no callback set 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3832728725 0 0) 0x7f8e940067e0 con 0x7f8e7c016be0 2024-03-25T19:28:56.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 10 client.4113.objecter ms_dispatch 0x7f8e9c000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:28:56.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:28:56.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:28:56.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-25T19:28:56.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8e94006c20 con 0x7f8e7c016be0 2024-03-25T19:28:56.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:56.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.326+0000 7f8e93fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:56.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.393+0000 7f8ea149a640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:28:56.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.393+0000 7f8ea149a640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:28:56.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.393+0000 7f8ea149a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:56.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.393+0000 7f8ea149a640 1 -- 172.21.15.19:0/2905489702 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8e9c0b3a70 con 0x7f8e7c016be0 2024-03-25T19:28:56.537 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:56.536+0000 7f68fe44b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:28:56.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.551+0000 7f8e93fff640 1 -- 172.21.15.19:0/2905489702 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f8e94005840 con 0x7f8e7c016be0 2024-03-25T19:28:56.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.551+0000 7f8e93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:28:56.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.551+0000 7f8e93fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:28:56.550 INFO:teuthology.orchestra.run.smithi019.stderr:adjusted tunables profile to default 2024-03-25T19:28:56.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.552+0000 7f8ea149a640 10 librados: watch_flush enter 2024-03-25T19:28:56.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.552+0000 7f8ea149a640 10 librados: watch_flush exit 2024-03-25T19:28:56.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.552+0000 7f8ea149a640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-25T19:28:56.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.552+0000 7f8ea149a640 10 client.4113.objecter successfully canceled tick 2024-03-25T19:28:56.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.552+0000 7f8ea149a640 10 monclient: shutdown 2024-03-25T19:28:56.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.552+0000 7f8ea149a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:56.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.552+0000 7f8ea149a640 1 -- 172.21.15.19:0/2905489702 >> v1:172.21.15.19:6789/0 conn(0x7f8e7c016be0 legacy=0x7f8e9c1e2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:56.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.552+0000 7f8ea149a640 1 -- 172.21.15.19:0/2905489702 shutdown_connections 2024-03-25T19:28:56.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.552+0000 7f8ea149a640 1 -- 172.21.15.19:0/2905489702 >> 172.21.15.19:0/2905489702 conn(0x7f8e9c0668e0 msgr2=0x7f8e9c0b69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:56.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.552+0000 7f8ea149a640 1 -- 172.21.15.19:0/2905489702 shutdown_connections 2024-03-25T19:28:56.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.552+0000 7f8ea149a640 1 -- 172.21.15.19:0/2905489702 wait complete. 2024-03-25T19:28:56.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:28:56.552+0000 7f8ea149a640 1 librados: shutdown 2024-03-25T19:28:56.563 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:28:56.563 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:28:56.563 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:56.563 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:28:56.573 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:56.573+0000 7fcf0f22b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:28:56.592 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:56.592 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:28:56.605 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:56.606+0000 7f68fe44b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:28:56.628 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:56.628+0000 7fcf0f22b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:28:56.631 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:56.631 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:28:56.699 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-25T19:28:56.699 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:28:56.769 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:56.769 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:28:56.796 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:56.796 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:28:56.864 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:56.864 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:28:56.931 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-03-25T19:28:56.932 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:28:56.999 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 2c96228b-453c-4e10-85b0-50a616f9e06b 0 2024-03-25T19:28:57.140 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.139+0000 7f604bc0f640 10 monclient: get_monmap_and_config 2024-03-25T19:28:57.140 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.139+0000 7f604bc0f640 10 monclient: build_initial_monmap 2024-03-25T19:28:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.139+0000 7f604bc0f640 10 monclient: monmap: 2024-03-25T19:28:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:57.141487+0000 2024-03-25T19:28:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:57.141487+0000 2024-03-25T19:28:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:28:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:28:57.141 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 1 Processor -- start 2024-03-25T19:28:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 1 -- start start 2024-03-25T19:28:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 10 monclient: init 2024-03-25T19:28:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 10 monclient(hunting): picked mon.noname-c con 0x7f60440b4700 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 10 monclient(hunting): picked mon.noname-a con 0x7f6044142a20 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 10 monclient(hunting): picked mon.noname-b con 0x7f60440a96b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60440a9d60 con 0x7f6044142a20 2024-03-25T19:28:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604413ff10 con 0x7f60440b4700 2024-03-25T19:28:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60440aa1f0 con 0x7f60440a96b0 2024-03-25T19:28:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.140+0000 7f604bc0f640 10 monclient(hunting): authenticate will time out at 5991.457641s 2024-03-25T19:28:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.141+0000 7f60499ab640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f60440b4700 0x7f6044142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:55614/0 (socket says 172.21.15.188:55614) 2024-03-25T19:28:57.144 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.141+0000 7f60499ab640 1 -- 172.21.15.188:0/3840923557 learned_addr learned my addr 172.21.15.188:0/3840923557 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:57.144 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.143+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171708891 0 0) 0x7f604413ff10 con 0x7f60440b4700 2024-03-25T19:28:57.144 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.143+0000 7f60489a9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.144 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.143+0000 7f60489a9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.144 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.143+0000 7f60489a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.144 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.143+0000 7f60489a9640 10 monclient(hunting): my global_id is 4111 2024-03-25T19:28:57.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.143+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6030003520 con 0x7f60440b4700 2024-03-25T19:28:57.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.143+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724808798 0 0) 0x7f60440aa1f0 con 0x7f60440a96b0 2024-03-25T19:28:57.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.143+0000 7f60489a9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.143+0000 7f60489a9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.143+0000 7f60489a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.143+0000 7f60489a9640 10 monclient(hunting): my global_id is 4109 2024-03-25T19:28:57.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.143+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f604413ff10 con 0x7f60440a96b0 2024-03-25T19:28:57.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.144+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808899538 0 0) 0x7f60440a9d60 con 0x7f6044142a20 2024-03-25T19:28:57.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.144+0000 7f60489a9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.144+0000 7f60489a9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.144+0000 7f60489a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.146 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.144+0000 7f60489a9640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:28:57.146 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.144+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60440aa1f0 con 0x7f6044142a20 2024-03-25T19:28:57.150 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 954152387 0 0) 0x7f60440aa1f0 con 0x7f6044142a20 2024-03-25T19:28:57.150 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 >> v1:172.21.15.19:6790/0 conn(0x7f60440a96b0 legacy=0x7f604413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.150 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 >> v1:172.21.15.188:6789/0 conn(0x7f60440b4700 legacy=0x7f6044142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.150 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 10 monclient: _finish_hunting 0 2024-03-25T19:28:57.150 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 1 monclient: found mon.noname-a 2024-03-25T19:28:57.150 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:57.150 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:28:57.150 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440b4ae0 con 0x7f6044142a20 2024-03-25T19:28:57.150 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 10 monclient: _finish_auth 0 2024-03-25T19:28:57.150 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 10 monclient: _check_auth_tickets 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f60340015b0 con 0x7f6044142a20 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f60489a9640 10 monclient: dump: 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:57.151 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.152 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f604bc0f640 5 monclient: authenticate success, global_id 4116 2024-03-25T19:28:57.152 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.149+0000 7f604bc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:57.152 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.151+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6034001710 con 0x7f6044142a20 2024-03-25T19:28:57.152 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.151+0000 7f60489a9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:57.152 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.151+0000 7f60489a9640 1 -- 172.21.15.188:0/3840923557 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f6034001b60 con 0x7f6044142a20 2024-03-25T19:28:57.152 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.151+0000 7f60489a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.152 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.151+0000 7f60489a9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:57.152 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.151+0000 7f60489a9640 10 monclient: dump: 2024-03-25T19:28:57.152 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:57.152 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:57.153 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.153 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.153 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.153 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:57.153 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:57.153 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:57.153 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:57.153 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.153 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.151+0000 7f604bc0f640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:57.153 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.151+0000 7f604bc0f640 4 set_mon_vals no callback set 2024-03-25T19:28:57.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.152+0000 7f604bc0f640 10 monclient: shutdown 2024-03-25T19:28:57.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.152+0000 7f604bc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:57.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.152+0000 7f604bc0f640 1 -- 172.21.15.188:0/3840923557 >> v1:172.21.15.19:6789/0 conn(0x7f6044142a20 legacy=0x7f604413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.152+0000 7f604bc0f640 1 -- 172.21.15.188:0/3840923557 shutdown_connections 2024-03-25T19:28:57.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.152+0000 7f604bc0f640 1 -- 172.21.15.188:0/3840923557 >> 172.21.15.188:0/3840923557 conn(0x7f60440668e0 msgr2=0x7f6044066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:57.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.152+0000 7f604bc0f640 1 -- 172.21.15.188:0/3840923557 shutdown_connections 2024-03-25T19:28:57.155 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.152+0000 7f604bc0f640 1 -- 172.21.15.188:0/3840923557 wait complete. 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 10 monclient: build_initial_monmap 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 10 monclient: monmap: 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 0.000000 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:created 0.000000 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 1 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 1 librados: starting msgr at 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 1 librados: starting objecter 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 1 Processor -- start 2024-03-25T19:28:57.156 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 1 -- start start 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 1 librados: setting wanted keys 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 1 librados: calling monclient init 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 10 monclient: init 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 10 monclient(hunting): picked mon.noname-c con 0x7f60440a96b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 10 monclient(hunting): picked mon.noname-a con 0x7f60440b4700 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 10 monclient(hunting): picked mon.noname-b con 0x7f6044142a20 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.153+0000 7f604bc0f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6044139150 con 0x7f60440b4700 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.154+0000 7f604bc0f640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60441393a0 con 0x7f6044142a20 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.154+0000 7f604bc0f640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60441381d0 con 0x7f60440a96b0 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.154+0000 7f604bc0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.154+0000 7f604bc0f640 10 monclient(hunting): authenticate will time out at 5991.470709s 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.154+0000 7f60491aa640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f60440b4700 0x7f604413c020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.188:53042/0 (socket says 172.21.15.188:53042) 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.154+0000 7f60491aa640 1 -- 172.21.15.188:0/552815658 learned_addr learned my addr 172.21.15.188:0/552815658 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.155+0000 7f603a7fc640 10 client.?.objecter ms_handle_connect 0x7f60440b4700 2024-03-25T19:28:57.157 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.155+0000 7f603a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:57.158 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.157+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226315901 0 0) 0x7f6044139150 con 0x7f60440b4700 2024-03-25T19:28:57.158 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.157+0000 7f603a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.158 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.157+0000 7f603a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.158 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.157+0000 7f603a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.158 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.157+0000 7f603a7fc640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:28:57.158 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.157+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6028004eb0 con 0x7f60440b4700 2024-03-25T19:28:57.161 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:57.161+0000 7fcf0f22b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:28:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.164+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2559262495 0 0) 0x7f6028004eb0 con 0x7f60440b4700 2024-03-25T19:28:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.164+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6044139150 con 0x7f60440b4700 2024-03-25T19:28:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.164+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f6034001b20 con 0x7f60440b4700 2024-03-25T19:28:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.164+0000 7f603a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.165 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.164+0000 7f603a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.164+0000 7f603a7fc640 10 monclient(hunting): dump: 2024-03-25T19:28:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:57.166 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.166 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.164+0000 7f603a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:57.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 681879423 0 0) 0x7f6044139150 con 0x7f60440b4700 2024-03-25T19:28:57.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 >> v1:172.21.15.19:6790/0 conn(0x7f60440a96b0 legacy=0x7f6044138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 >> v1:172.21.15.188:6789/0 conn(0x7f6044142a20 legacy=0x7f60441daa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f603a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:57.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f603a7fc640 1 monclient: found mon.a 2024-03-25T19:28:57.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f603a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f603a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6044138360 con 0x7f60440b4700 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f603a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f603a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f603a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f604bc0f640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f604bc0f640 10 monclient: _renew_subs 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f604bc0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f604bc0f640 1 -- 172.21.15.188:0/552815658 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6044135930 con 0x7f60440b4700 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f604bc0f640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f604bc0f640 10 monclient: _renew_subs 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f604bc0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f604bc0f640 1 -- 172.21.15.188:0/552815658 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f604413c910 con 0x7f60440b4700 2024-03-25T19:28:57.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.169+0000 7f604bc0f640 1 librados: init done 2024-03-25T19:28:57.172 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.170+0000 7f604bc0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:57.172 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.170+0000 7f604bc0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:57.172 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.170+0000 7f604bc0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:57.172 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.170+0000 7f604bc0f640 1 -- 172.21.15.188:0/552815658 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604413cc10 con 0x7f60440b4700 2024-03-25T19:28:57.217 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:57.217+0000 7fcf0f22b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:28:57.258 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:57.259+0000 7f68fe44b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:28:57.262 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:57.262+0000 7fcf0f22b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:28:57.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.308+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60340038a0 con 0x7f60440b4700 2024-03-25T19:28:57.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.308+0000 7f603a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:57.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.308+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f6034003cf0 con 0x7f60440b4700 2024-03-25T19:28:57.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.308+0000 7f603a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.308+0000 7f603a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.308+0000 7f603a7fc640 10 monclient: dump: 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.308+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f60340042a0 con 0x7f60440b4700 2024-03-25T19:28:57.310 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.308+0000 7f603bfff640 4 set_mon_vals no callback set 2024-03-25T19:28:57.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.309+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2245430019 0 0) 0x7f6034006810 con 0x7f60440b4700 2024-03-25T19:28:57.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.309+0000 7f603a7fc640 10 client.4119.objecter ms_dispatch 0x7f6044000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:28:57.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.309+0000 7f603a7fc640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:28:57.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.309+0000 7f603a7fc640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:28:57.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.309+0000 7f603a7fc640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:28:57.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.309+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6034006a80 con 0x7f60440b4700 2024-03-25T19:28:57.312 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.309+0000 7f603a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:57.312 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.309+0000 7f603a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:57.315 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:57.316+0000 7f68fe44b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:28:57.315 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:57.315+0000 7fcf0f22b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:28:57.361 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:57.362+0000 7f68fe44b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:28:57.362 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:57.359+0000 7fcf0f22b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:28:57.383 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.382+0000 7f604bc0f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2c96228b-453c-4e10-85b0-50a616f9e06b", "id": 0}] 2024-03-25T19:28:57.383 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.382+0000 7f604bc0f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2c96228b-453c-4e10-85b0-50a616f9e06b", "id": 0}] 2024-03-25T19:28:57.383 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.382+0000 7f604bc0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:57.384 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.383+0000 7f604bc0f640 1 -- 172.21.15.188:0/552815658 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2c96228b-453c-4e10-85b0-50a616f9e06b", "id": 0} v 0) v1 -- 0x7f604404f730 con 0x7f60440b4700 2024-03-25T19:28:57.418 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:57.419+0000 7f68fe44b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:28:57.463 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:57.465+0000 7f68fe44b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:28:57.476 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:57.476+0000 7fcf0f22b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:28:57.491 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.490+0000 7f603a7fc640 1 -- 172.21.15.188:0/552815658 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2c96228b-453c-4e10-85b0-50a616f9e06b", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1789672434 0 732707515) 0x7f6034005900 con 0x7f60440b4700 2024-03-25T19:28:57.491 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.491+0000 7f603a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2c96228b-453c-4e10-85b0-50a616f9e06b", "id": 0}] 2024-03-25T19:28:57.491 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.491+0000 7f603a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:57.492 INFO:teuthology.orchestra.run.smithi188.stdout:0 2024-03-25T19:28:57.493 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.491+0000 7f604bc0f640 10 librados: watch_flush enter 2024-03-25T19:28:57.493 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.491+0000 7f604bc0f640 10 librados: watch_flush exit 2024-03-25T19:28:57.493 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.491+0000 7f604bc0f640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:28:57.493 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.491+0000 7f604bc0f640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:28:57.493 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.491+0000 7f604bc0f640 10 monclient: shutdown 2024-03-25T19:28:57.493 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.491+0000 7f604bc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:57.493 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.491+0000 7f604bc0f640 1 -- 172.21.15.188:0/552815658 >> v1:172.21.15.19:6789/0 conn(0x7f60440b4700 legacy=0x7f604413c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.493 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.492+0000 7f604bc0f640 1 -- 172.21.15.188:0/552815658 shutdown_connections 2024-03-25T19:28:57.493 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.492+0000 7f604bc0f640 1 -- 172.21.15.188:0/552815658 >> 172.21.15.188:0/552815658 conn(0x7f60440668e0 msgr2=0x7f60440b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:57.493 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.492+0000 7f604bc0f640 1 -- 172.21.15.188:0/552815658 shutdown_connections 2024-03-25T19:28:57.493 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.492+0000 7f604bc0f640 1 -- 172.21.15.188:0/552815658 wait complete. 2024-03-25T19:28:57.494 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.492+0000 7f604bc0f640 1 librados: shutdown 2024-03-25T19:28:57.503 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 52a1fe90-7f0c-443d-b654-68ad85b9d9ee 1 2024-03-25T19:28:57.553 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:57.552+0000 7fcf0f22b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:28:57.587 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:57.589+0000 7f68fe44b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:28:57.601 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:57.601+0000 7fcf0f22b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:28:57.604 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.604+0000 7fd7caad6640 10 monclient: get_monmap_and_config 2024-03-25T19:28:57.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.604+0000 7fd7caad6640 10 monclient: build_initial_monmap 2024-03-25T19:28:57.605 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.604+0000 7fd7caad6640 10 monclient: monmap: 2024-03-25T19:28:57.605 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:57.605 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:57.605 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:57.606170+0000 2024-03-25T19:28:57.605 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:57.606170+0000 2024-03-25T19:28:57.605 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:57.605 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:57.605 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:57.605 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:28:57.605 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:28:57.605 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 1 Processor -- start 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 1 -- start start 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 10 monclient: init 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c40b4700 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c4142a20 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 10 monclient(hunting): picked mon.noname-c con 0x7fd7c40a96b0 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40a9d60 con 0x7fd7c40b4700 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c413ff10 con 0x7fd7c40a96b0 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40aa1f0 con 0x7fd7c4142a20 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:57.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.605+0000 7fd7caad6640 10 monclient(hunting): authenticate will time out at 5991.922325s 2024-03-25T19:28:57.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.606+0000 7fd7c9073640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fd7c40a96b0 0x7fd7c413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:55622/0 (socket says 172.21.15.188:55622) 2024-03-25T19:28:57.608 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.606+0000 7fd7c9073640 1 -- 172.21.15.188:0/4095598977 learned_addr learned my addr 172.21.15.188:0/4095598977 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.608+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052083219 0 0) 0x7fd7c413ff10 con 0x7fd7c40a96b0 2024-03-25T19:28:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.608+0000 7fd7c37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.608+0000 7fd7c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.608+0000 7fd7c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.608+0000 7fd7c37fe640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:28:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.608+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b4003520 con 0x7fd7c40a96b0 2024-03-25T19:28:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.608+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654474652 0 0) 0x7fd7c40a9d60 con 0x7fd7c40b4700 2024-03-25T19:28:57.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.608+0000 7fd7c37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.610 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.608+0000 7fd7c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.610 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.608+0000 7fd7c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.610 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.608+0000 7fd7c37fe640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:28:57.610 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.608+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c413ff10 con 0x7fd7c40b4700 2024-03-25T19:28:57.610 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.609+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732958204 0 0) 0x7fd7c40aa1f0 con 0x7fd7c4142a20 2024-03-25T19:28:57.610 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.609+0000 7fd7c37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.610 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.609+0000 7fd7c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.611 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.609+0000 7fd7c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.611 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.609+0000 7fd7c37fe640 10 monclient(hunting): my global_id is 4112 2024-03-25T19:28:57.611 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.609+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c40a9d60 con 0x7fd7c4142a20 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285638966 0 0) 0x7fd7b4003520 con 0x7fd7c40a96b0 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 >> v1:172.21.15.19:6790/0 conn(0x7fd7c4142a20 legacy=0x7fd7c413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 >> v1:172.21.15.19:6789/0 conn(0x7fd7c40b4700 legacy=0x7fd7c4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 1 monclient: found mon.noname-c 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c40b4ae0 con 0x7fd7c40a96b0 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 10 monclient: _finish_auth 0 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd7bc001a10 con 0x7fd7c40a96b0 2024-03-25T19:28:57.613 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7caad6640 5 monclient: authenticate success, global_id 4114 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7caad6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7c37fe640 10 monclient: dump: 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:57.614 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.615 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.612+0000 7fd7caad6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:57.639 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.638+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7bc001b70 con 0x7fd7c40a96b0 2024-03-25T19:28:57.639 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.638+0000 7fd7c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:57.639 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.638+0000 7fd7c37fe640 1 -- 172.21.15.188:0/4095598977 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd7bc001fc0 con 0x7fd7c40a96b0 2024-03-25T19:28:57.639 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.638+0000 7fd7caad6640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:57.639 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.638+0000 7fd7caad6640 4 set_mon_vals no callback set 2024-03-25T19:28:57.639 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.638+0000 7fd7c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.639 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.638+0000 7fd7c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:57.639 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.638+0000 7fd7c37fe640 10 monclient: dump: 2024-03-25T19:28:57.639 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:57.640 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:57.640 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.640 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.640 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.640 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:57.640 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:57.640 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:57.640 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:57.640 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.643 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.639+0000 7fd7caad6640 10 monclient: shutdown 2024-03-25T19:28:57.643 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.639+0000 7fd7caad6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:57.643 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.639+0000 7fd7caad6640 1 -- 172.21.15.188:0/4095598977 >> v1:172.21.15.188:6789/0 conn(0x7fd7c40a96b0 legacy=0x7fd7c413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.643 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.639+0000 7fd7caad6640 1 -- 172.21.15.188:0/4095598977 shutdown_connections 2024-03-25T19:28:57.643 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.639+0000 7fd7caad6640 1 -- 172.21.15.188:0/4095598977 >> 172.21.15.188:0/4095598977 conn(0x7fd7c40668e0 msgr2=0x7fd7c4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:57.643 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.639+0000 7fd7caad6640 1 -- 172.21.15.188:0/4095598977 shutdown_connections 2024-03-25T19:28:57.643 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.639+0000 7fd7caad6640 1 -- 172.21.15.188:0/4095598977 wait complete. 2024-03-25T19:28:57.643 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.639+0000 7fd7caad6640 10 monclient: build_initial_monmap 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.639+0000 7fd7caad6640 10 monclient: monmap: 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 0.000000 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:created 0.000000 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 1 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.640+0000 7fd7caad6640 1 librados: starting msgr at 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.640+0000 7fd7caad6640 1 librados: starting objecter 2024-03-25T19:28:57.644 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.640+0000 7fd7caad6640 1 Processor -- start 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.640+0000 7fd7caad6640 1 -- start start 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.640+0000 7fd7caad6640 1 librados: setting wanted keys 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.640+0000 7fd7caad6640 1 librados: calling monclient init 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.640+0000 7fd7caad6640 10 monclient: init 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.640+0000 7fd7caad6640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.640+0000 7fd7caad6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.640+0000 7fd7caad6640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c40a96b0 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.641+0000 7fd7caad6640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c40b4700 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.641+0000 7fd7caad6640 10 monclient(hunting): picked mon.noname-c con 0x7fd7c4142a20 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.641+0000 7fd7caad6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c412ebc0 con 0x7fd7c40a96b0 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.641+0000 7fd7caad6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40adf90 con 0x7fd7c40b4700 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.641+0000 7fd7caad6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40ae200 con 0x7fd7c4142a20 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.641+0000 7fd7caad6640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:57.645 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.641+0000 7fd7caad6640 10 monclient(hunting): authenticate will time out at 5991.957749s 2024-03-25T19:28:57.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.641+0000 7fd7c9073640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fd7c4142a20 0x7fd7c412f740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.188:57894/0 (socket says 172.21.15.188:57894) 2024-03-25T19:28:57.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.641+0000 7fd7c9073640 1 -- 172.21.15.188:0/1091025476 learned_addr learned my addr 172.21.15.188:0/1091025476 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:57.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.641+0000 7fd7c17fa640 10 client.?.objecter ms_handle_connect 0x7fd7c4142a20 2024-03-25T19:28:57.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.642+0000 7fd7c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:57.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.642+0000 7fd7c17fa640 10 client.?.objecter ms_handle_connect 0x7fd7c40a96b0 2024-03-25T19:28:57.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.642+0000 7fd7c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:57.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.642+0000 7fd7c17fa640 10 client.?.objecter ms_handle_connect 0x7fd7c40b4700 2024-03-25T19:28:57.646 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.642+0000 7fd7c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:57.647 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065083954 0 0) 0x7fd7c412ebc0 con 0x7fd7c40a96b0 2024-03-25T19:28:57.647 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.647 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.647 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.647 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:28:57.647 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a8004eb0 con 0x7fd7c40a96b0 2024-03-25T19:28:57.647 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411138267 0 0) 0x7fd7c40ae200 con 0x7fd7c4142a20 2024-03-25T19:28:57.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:28:57.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c412ebc0 con 0x7fd7c4142a20 2024-03-25T19:28:57.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308289515 0 0) 0x7fd7c40adf90 con 0x7fd7c40b4700 2024-03-25T19:28:57.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:28:57.648 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.644+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c40ae200 con 0x7fd7c40b4700 2024-03-25T19:28:57.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.649+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3709172007 0 0) 0x7fd7a8004eb0 con 0x7fd7c40a96b0 2024-03-25T19:28:57.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.649+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7c40adf90 con 0x7fd7c40a96b0 2024-03-25T19:28:57.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.649+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd7b8001a50 con 0x7fd7c40a96b0 2024-03-25T19:28:57.650 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.649+0000 7fd7c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.649+0000 7fd7c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.649+0000 7fd7c17fa640 10 monclient(hunting): dump: 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.651 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.649+0000 7fd7c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:57.654 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3636881515 0 0) 0x7fd7c40adf90 con 0x7fd7c40a96b0 2024-03-25T19:28:57.654 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 >> v1:172.21.15.19:6790/0 conn(0x7fd7c4142a20 legacy=0x7fd7c412f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.654 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 >> v1:172.21.15.188:6789/0 conn(0x7fd7c40b4700 legacy=0x7fd7c412f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.654 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7c17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:28:57.654 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7c17fa640 1 monclient: found mon.a 2024-03-25T19:28:57.654 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:57.654 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7c17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c40ac0b0 con 0x7fd7c40a96b0 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7c17fa640 10 monclient: _finish_auth 0 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7c17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7caad6640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7caad6640 10 monclient: _renew_subs 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7caad6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7caad6640 1 -- 172.21.15.188:0/1091025476 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c40acc90 con 0x7fd7c40a96b0 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7caad6640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7caad6640 10 monclient: _renew_subs 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7caad6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7caad6640 1 -- 172.21.15.188:0/1091025476 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c41e37e0 con 0x7fd7c40a96b0 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.652+0000 7fd7caad6640 1 librados: init done 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.653+0000 7fd7caad6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:57.655 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.653+0000 7fd7caad6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:57.656 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.653+0000 7fd7caad6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:57.656 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.653+0000 7fd7caad6640 1 -- 172.21.15.188:0/1091025476 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c40b7070 con 0x7fd7c40a96b0 2024-03-25T19:28:57.658 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:57.657+0000 7fcf0f22b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:28:57.665 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:57.666+0000 7f68fe44b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:28:57.706 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:57.706+0000 7fcf0f22b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:28:57.714 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:57.715+0000 7f68fe44b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:28:57.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.759+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7b80035f0 con 0x7fd7c40a96b0 2024-03-25T19:28:57.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.759+0000 7fd7c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:57.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.759+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd7b8003900 con 0x7fd7c40a96b0 2024-03-25T19:28:57.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.759+0000 7fd7c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.759+0000 7fd7c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:57.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.759+0000 7fd7c17fa640 10 monclient: dump: 2024-03-25T19:28:57.760 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:57.761 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:57.761 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.761 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.761 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.761 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:57.761 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:57.761 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:57.761 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:57.761 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.761 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.759+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd7b8003eb0 con 0x7fd7c40a96b0 2024-03-25T19:28:57.761 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.759+0000 7fd7c2ffd640 4 set_mon_vals no callback set 2024-03-25T19:28:57.762 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.761+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3136460222 0 0) 0x7fd7b80054e0 con 0x7fd7c40a96b0 2024-03-25T19:28:57.762 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.761+0000 7fd7c17fa640 10 client.4125.objecter ms_dispatch 0x7fd7c4000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:28:57.762 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.761+0000 7fd7c17fa640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:28:57.762 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.761+0000 7fd7c17fa640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:28:57.762 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.761+0000 7fd7c17fa640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:28:57.762 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.761+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd7b8005830 con 0x7fd7c40a96b0 2024-03-25T19:28:57.763 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.761+0000 7fd7c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:57.763 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.761+0000 7fd7c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:57.769 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:57.771+0000 7f68fe44b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:28:57.818 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:57.820+0000 7f68fe44b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:28:57.835 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.834+0000 7fd7caad6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "52a1fe90-7f0c-443d-b654-68ad85b9d9ee", "id": 1}] 2024-03-25T19:28:57.835 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.834+0000 7fd7caad6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "52a1fe90-7f0c-443d-b654-68ad85b9d9ee", "id": 1}] 2024-03-25T19:28:57.835 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.834+0000 7fd7caad6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:57.835 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.834+0000 7fd7caad6640 1 -- 172.21.15.188:0/1091025476 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd new", "uuid": "52a1fe90-7f0c-443d-b654-68ad85b9d9ee", "id": 1} v 0) v1 -- 0x7fd7c40b3df0 con 0x7fd7c40a96b0 2024-03-25T19:28:57.870 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.869+0000 7fd7c17fa640 1 -- 172.21.15.188:0/1091025476 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "52a1fe90-7f0c-443d-b654-68ad85b9d9ee", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1070171414 0 940483276) 0x7fd7b8005a50 con 0x7fd7c40a96b0 2024-03-25T19:28:57.870 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.869+0000 7fd7c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "52a1fe90-7f0c-443d-b654-68ad85b9d9ee", "id": 1}] 2024-03-25T19:28:57.870 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.869+0000 7fd7c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:57.870 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-03-25T19:28:57.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.870+0000 7fd7caad6640 10 librados: watch_flush enter 2024-03-25T19:28:57.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.870+0000 7fd7caad6640 10 librados: watch_flush exit 2024-03-25T19:28:57.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.870+0000 7fd7caad6640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:28:57.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.870+0000 7fd7caad6640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:28:57.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.870+0000 7fd7caad6640 10 monclient: shutdown 2024-03-25T19:28:57.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.870+0000 7fd7caad6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:57.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.870+0000 7fd7caad6640 1 -- 172.21.15.188:0/1091025476 >> v1:172.21.15.19:6789/0 conn(0x7fd7c40a96b0 legacy=0x7fd7c40ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.870+0000 7fd7caad6640 1 -- 172.21.15.188:0/1091025476 shutdown_connections 2024-03-25T19:28:57.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.870+0000 7fd7caad6640 1 -- 172.21.15.188:0/1091025476 >> 172.21.15.188:0/1091025476 conn(0x7fd7c40668e0 msgr2=0x7fd7c413f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:57.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.871+0000 7fd7caad6640 1 -- 172.21.15.188:0/1091025476 shutdown_connections 2024-03-25T19:28:57.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.871+0000 7fd7caad6640 1 -- 172.21.15.188:0/1091025476 wait complete. 2024-03-25T19:28:57.872 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.871+0000 7fd7caad6640 1 librados: shutdown 2024-03-25T19:28:57.883 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new a8b5725b-de23-4b36-83b4-4276ca95aad7 2 2024-03-25T19:28:57.984 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.983+0000 7f36ce8e6640 10 monclient: get_monmap_and_config 2024-03-25T19:28:57.984 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.983+0000 7f36ce8e6640 10 monclient: build_initial_monmap 2024-03-25T19:28:57.984 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.983+0000 7f36ce8e6640 10 monclient: monmap: 2024-03-25T19:28:57.984 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:57.984 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:57.984 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:57.985137+0000 2024-03-25T19:28:57.984 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:57.985137+0000 2024-03-25T19:28:57.984 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:57.984 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:57.984 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:57.984 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:28:57.985 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:28:57.985 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.986 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.983+0000 7f36ce8e6640 1 Processor -- start 2024-03-25T19:28:57.986 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.984+0000 7f36ce8e6640 1 -- start start 2024-03-25T19:28:57.986 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.984+0000 7f36ce8e6640 10 monclient: init 2024-03-25T19:28:57.986 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.984+0000 7f36ce8e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:57.986 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.984+0000 7f36ce8e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:57.986 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.984+0000 7f36ce8e6640 10 monclient(hunting): picked mon.noname-a con 0x7f36c80b4700 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:57.986 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.984+0000 7f36ce8e6640 10 monclient(hunting): picked mon.noname-b con 0x7f36c8142a20 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:57.986 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.984+0000 7f36ce8e6640 10 monclient(hunting): picked mon.noname-c con 0x7f36c80a96b0 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:57.986 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.984+0000 7f36ce8e6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c80a9d60 con 0x7f36c80b4700 2024-03-25T19:28:57.987 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.984+0000 7f36ce8e6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c813ff10 con 0x7f36c80a96b0 2024-03-25T19:28:57.987 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.984+0000 7f36ce8e6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c80aa1f0 con 0x7f36c8142a20 2024-03-25T19:28:57.987 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.984+0000 7f36ce8e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:57.987 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.984+0000 7f36ce8e6640 10 monclient(hunting): authenticate will time out at 5992.301325s 2024-03-25T19:28:57.987 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.985+0000 7f36cce83640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f36c80a96b0 0x7f36c813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:55646/0 (socket says 172.21.15.188:55646) 2024-03-25T19:28:57.987 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.985+0000 7f36cce83640 1 -- 172.21.15.188:0/3805654644 learned_addr learned my addr 172.21.15.188:0/3805654644 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:57.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.987+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309785771 0 0) 0x7f36c80a9d60 con 0x7f36c80b4700 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.987+0000 7f36c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.987+0000 7f36c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.987+0000 7f36c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.987+0000 7f36c6ffd640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.987+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b0003520 con 0x7f36c80b4700 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.987+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620750033 0 0) 0x7f36c813ff10 con 0x7f36c80a96b0 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.987+0000 7f36c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.987+0000 7f36c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.987+0000 7f36c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.987+0000 7f36c6ffd640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.987+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c80a9d60 con 0x7f36c80a96b0 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.988+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371700548 0 0) 0x7f36c80aa1f0 con 0x7f36c8142a20 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.988+0000 7f36c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.988+0000 7f36c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.990 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.988+0000 7f36c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.990 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.988+0000 7f36c6ffd640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:28:57.990 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.988+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c813ff10 con 0x7f36c8142a20 2024-03-25T19:28:57.992 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3691310010 0 0) 0x7f36b0003520 con 0x7f36c80b4700 2024-03-25T19:28:57.992 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 >> v1:172.21.15.19:6790/0 conn(0x7f36c8142a20 legacy=0x7f36c813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.992 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 >> v1:172.21.15.188:6789/0 conn(0x7f36c80a96b0 legacy=0x7f36c813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.992 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:57.992 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:28:57.992 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:57.992 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c80b4ae0 con 0x7f36c80b4700 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f36bc001690 con 0x7f36c80b4700 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36c6ffd640 10 monclient: dump: 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:57.993 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:57.994 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:57.994 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:57.994 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:57.994 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36ce8e6640 5 monclient: authenticate success, global_id 4128 2024-03-25T19:28:57.994 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:57.991+0000 7f36ce8e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:58.000 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36bc0017f0 con 0x7f36c80b4700 2024-03-25T19:28:58.000 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:58.000 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36c6ffd640 1 -- 172.21.15.188:0/3805654644 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f36bc001c00 con 0x7f36c80b4700 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36ce8e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36ce8e6640 4 set_mon_vals no callback set 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36c6ffd640 10 monclient: dump: 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.001 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.004 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36ce8e6640 10 monclient: shutdown 2024-03-25T19:28:58.004 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36ce8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:58.004 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36ce8e6640 1 -- 172.21.15.188:0/3805654644 >> v1:172.21.15.19:6789/0 conn(0x7f36c80b4700 legacy=0x7f36c8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.004 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36ce8e6640 1 -- 172.21.15.188:0/3805654644 shutdown_connections 2024-03-25T19:28:58.004 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36ce8e6640 1 -- 172.21.15.188:0/3805654644 >> 172.21.15.188:0/3805654644 conn(0x7f36c80668e0 msgr2=0x7f36c8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:58.004 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36ce8e6640 1 -- 172.21.15.188:0/3805654644 shutdown_connections 2024-03-25T19:28:58.004 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.000+0000 7f36ce8e6640 1 -- 172.21.15.188:0/3805654644 wait complete. 2024-03-25T19:28:58.004 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.001+0000 7f36ce8e6640 10 monclient: build_initial_monmap 2024-03-25T19:28:58.004 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.001+0000 7f36ce8e6640 10 monclient: monmap: 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 0.000000 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:created 0.000000 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 1 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.001+0000 7f36ce8e6640 1 librados: starting msgr at 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.001+0000 7f36ce8e6640 1 librados: starting objecter 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.001+0000 7f36ce8e6640 1 Processor -- start 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 1 -- start start 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 1 librados: setting wanted keys 2024-03-25T19:28:58.005 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 1 librados: calling monclient init 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 10 monclient: init 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 10 monclient(hunting): picked mon.noname-c con 0x7f36c80a96b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 10 monclient(hunting): picked mon.noname-a con 0x7f36c80b4700 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 10 monclient(hunting): picked mon.noname-b con 0x7f36c8142a20 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c80ad6b0 con 0x7f36c80b4700 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c80ad920 con 0x7f36c8142a20 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c80adc00 con 0x7f36c80a96b0 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.002+0000 7f36ce8e6640 10 monclient(hunting): authenticate will time out at 5992.319424s 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.003+0000 7f36c77fe640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f36c80b4700 0x7f36c80ae3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.188:53066/0 (socket says 172.21.15.188:53066) 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.003+0000 7f36c77fe640 1 -- 172.21.15.188:0/4147722574 learned_addr learned my addr 172.21.15.188:0/4147722574 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:58.006 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.003+0000 7f36c4ff9640 10 client.?.objecter ms_handle_connect 0x7f36c80b4700 2024-03-25T19:28:58.007 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.003+0000 7f36c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:58.007 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.005+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708757845 0 0) 0x7f36c80ad6b0 con 0x7f36c80b4700 2024-03-25T19:28:58.007 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.005+0000 7f36c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.007 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.005+0000 7f36c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.007 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.005+0000 7f36c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.007 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.005+0000 7f36c4ff9640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:28:58.007 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.005+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b4004eb0 con 0x7f36c80b4700 2024-03-25T19:28:58.010 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.010+0000 7f36c4ff9640 10 client.?.objecter ms_handle_connect 0x7f36c80a96b0 2024-03-25T19:28:58.010 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.010+0000 7f36c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:58.011 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.010+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1037993191 0 0) 0x7f36b4004eb0 con 0x7f36c80b4700 2024-03-25T19:28:58.011 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.010+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36c80ad6b0 con 0x7f36c80b4700 2024-03-25T19:28:58.011 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.010+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f36b8001ad0 con 0x7f36c80b4700 2024-03-25T19:28:58.011 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.010+0000 7f36c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.011 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.010+0000 7f36c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:58.011 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.010+0000 7f36c4ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:58.012 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.012 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.012 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.012 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.012 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.012 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.012 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.012 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.012 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.012 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.012 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.010+0000 7f36c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:58.013 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.012+0000 7f36c4ff9640 10 client.?.objecter ms_handle_connect 0x7f36c8142a20 2024-03-25T19:28:58.013 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.012+0000 7f36c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:58.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 754754644 0 0) 0x7f36c80ad6b0 con 0x7f36c80b4700 2024-03-25T19:28:58.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 >> v1:172.21.15.19:6790/0 conn(0x7f36c80a96b0 legacy=0x7f36c813a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 >> v1:172.21.15.188:6789/0 conn(0x7f36c8142a20 legacy=0x7f36c80ac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36c4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:28:58.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36c4ff9640 1 monclient: found mon.a 2024-03-25T19:28:58.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:58.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36c4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c80acc30 con 0x7f36c80b4700 2024-03-25T19:28:58.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36c4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:28:58.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36c4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:28:58.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:58.017 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36ce8e6640 5 monclient: authenticate success, global_id 4131 2024-03-25T19:28:58.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36ce8e6640 10 monclient: _renew_subs 2024-03-25T19:28:58.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36ce8e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.015+0000 7f36ce8e6640 1 -- 172.21.15.188:0/4147722574 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c80ab880 con 0x7f36c80b4700 2024-03-25T19:28:58.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.016+0000 7f36ce8e6640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:58.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.016+0000 7f36ce8e6640 10 monclient: _renew_subs 2024-03-25T19:28:58.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.016+0000 7f36ce8e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.016+0000 7f36ce8e6640 1 -- 172.21.15.188:0/4147722574 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c81e3270 con 0x7f36c80b4700 2024-03-25T19:28:58.018 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.016+0000 7f36ce8e6640 1 librados: init done 2024-03-25T19:28:58.019 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.016+0000 7f36ce8e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:58.019 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.016+0000 7f36ce8e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:58.019 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.016+0000 7f36ce8e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.019 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.016+0000 7f36ce8e6640 1 -- 172.21.15.188:0/4147722574 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c80aca70 con 0x7f36c80b4700 2024-03-25T19:28:58.109 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:58.108+0000 7fcf0f22b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:28:58.126 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.125+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36b8001c30 con 0x7f36c80b4700 2024-03-25T19:28:58.126 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.125+0000 7f36c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:58.126 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.125+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f36b8003ae0 con 0x7f36c80b4700 2024-03-25T19:28:58.126 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.125+0000 7f36c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.126 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.125+0000 7f36c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:58.126 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.125+0000 7f36c4ff9640 10 monclient: dump: 2024-03-25T19:28:58.126 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.126 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.126 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.126 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.126 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.126 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.127 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.127 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.127 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.127 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.127 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.125+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f36b8004090 con 0x7f36c80b4700 2024-03-25T19:28:58.127 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.125+0000 7f36c5ffb640 4 set_mon_vals no callback set 2024-03-25T19:28:58.128 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.127+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 4037344450 0 0) 0x7f36b8006780 con 0x7f36c80b4700 2024-03-25T19:28:58.128 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.127+0000 7f36c4ff9640 10 client.4131.objecter ms_dispatch 0x7f36c8000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:28:58.128 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.127+0000 7f36c4ff9640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:28:58.128 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.127+0000 7f36c4ff9640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:28:58.128 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.127+0000 7f36c4ff9640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T19:28:58.128 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.127+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f36b8006360 con 0x7f36c80b4700 2024-03-25T19:28:58.128 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.127+0000 7f36c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:58.128 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.127+0000 7f36c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:58.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.201+0000 7f36ce8e6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a8b5725b-de23-4b36-83b4-4276ca95aad7", "id": 2}] 2024-03-25T19:28:58.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.201+0000 7f36ce8e6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a8b5725b-de23-4b36-83b4-4276ca95aad7", "id": 2}] 2024-03-25T19:28:58.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.201+0000 7f36ce8e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.201+0000 7f36ce8e6640 1 -- 172.21.15.188:0/4147722574 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a8b5725b-de23-4b36-83b4-4276ca95aad7", "id": 2} v 0) v1 -- 0x7f36c804f730 con 0x7f36c80b4700 2024-03-25T19:28:58.228 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:58.230+0000 7f68fe44b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:28:58.245 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.244+0000 7f36c4ff9640 1 -- 172.21.15.188:0/4147722574 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a8b5725b-de23-4b36-83b4-4276ca95aad7", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3730625809 0 216599125) 0x7f36b8006a70 con 0x7f36c80b4700 2024-03-25T19:28:58.245 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.244+0000 7f36c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a8b5725b-de23-4b36-83b4-4276ca95aad7", "id": 2}] 2024-03-25T19:28:58.245 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.244+0000 7f36c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:58.245 INFO:teuthology.orchestra.run.smithi188.stdout:2 2024-03-25T19:28:58.246 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.244+0000 7f36ce8e6640 10 librados: watch_flush enter 2024-03-25T19:28:58.246 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.244+0000 7f36ce8e6640 10 librados: watch_flush exit 2024-03-25T19:28:58.246 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.244+0000 7f36ce8e6640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T19:28:58.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.244+0000 7f36ce8e6640 10 client.4131.objecter successfully canceled tick 2024-03-25T19:28:58.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.245+0000 7f36ce8e6640 10 monclient: shutdown 2024-03-25T19:28:58.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.245+0000 7f36ce8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:58.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.245+0000 7f36ce8e6640 1 -- 172.21.15.188:0/4147722574 >> v1:172.21.15.19:6789/0 conn(0x7f36c80b4700 legacy=0x7f36c80ae3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.245+0000 7f36ce8e6640 1 -- 172.21.15.188:0/4147722574 shutdown_connections 2024-03-25T19:28:58.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.245+0000 7f36ce8e6640 1 -- 172.21.15.188:0/4147722574 >> 172.21.15.188:0/4147722574 conn(0x7f36c80668e0 msgr2=0x7f36c813c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:58.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.245+0000 7f36ce8e6640 1 -- 172.21.15.188:0/4147722574 shutdown_connections 2024-03-25T19:28:58.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.245+0000 7f36ce8e6640 1 -- 172.21.15.188:0/4147722574 wait complete. 2024-03-25T19:28:58.247 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.245+0000 7f36ce8e6640 1 librados: shutdown 2024-03-25T19:28:58.256 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 8f6cea59-df77-45e1-960e-96fcfe4f78b4 3 2024-03-25T19:28:58.275 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:58.275+0000 7fcf0f22b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:28:58.357 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.356+0000 7fad653c8640 10 monclient: get_monmap_and_config 2024-03-25T19:28:58.357 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.356+0000 7fad653c8640 10 monclient: build_initial_monmap 2024-03-25T19:28:58.357 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.356+0000 7fad653c8640 10 monclient: monmap: 2024-03-25T19:28:58.357 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:58.357 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.357 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:58.358099+0000 2024-03-25T19:28:58.357 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:58.358099+0000 2024-03-25T19:28:58.357 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:58.357 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.357 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:58.357 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:28:58.357 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:28:58.358 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.359 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.356+0000 7fad653c8640 1 Processor -- start 2024-03-25T19:28:58.359 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.356+0000 7fad653c8640 1 -- start start 2024-03-25T19:28:58.359 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.356+0000 7fad653c8640 10 monclient: init 2024-03-25T19:28:58.359 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.357+0000 7fad653c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:58.359 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.357+0000 7fad653c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:58.359 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.357+0000 7fad653c8640 10 monclient(hunting): picked mon.noname-b con 0x7fad600b4700 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:58.359 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.357+0000 7fad653c8640 10 monclient(hunting): picked mon.noname-c con 0x7fad60142a20 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:58.359 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.357+0000 7fad653c8640 10 monclient(hunting): picked mon.noname-a con 0x7fad600a96b0 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:58.359 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.357+0000 7fad653c8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad600a9d60 con 0x7fad600a96b0 2024-03-25T19:28:58.359 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.357+0000 7fad653c8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6013ff10 con 0x7fad60142a20 2024-03-25T19:28:58.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.357+0000 7fad653c8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad600aa1f0 con 0x7fad600b4700 2024-03-25T19:28:58.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.357+0000 7fad653c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:58.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.357+0000 7fad653c8640 10 monclient(hunting): authenticate will time out at 5992.674199s 2024-03-25T19:28:58.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.357+0000 7fad5e7fc640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fad60142a20 0x7fad6013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:55664/0 (socket says 172.21.15.188:55664) 2024-03-25T19:28:58.360 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.357+0000 7fad5e7fc640 1 -- 172.21.15.188:0/3398428500 learned_addr learned my addr 172.21.15.188:0/3398428500 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:58.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.360+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067850236 0 0) 0x7fad600a9d60 con 0x7fad600a96b0 2024-03-25T19:28:58.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.360+0000 7fad5dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.360+0000 7fad5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.360+0000 7fad5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.360+0000 7fad5dffb640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:28:58.361 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.360+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad48003520 con 0x7fad600a96b0 2024-03-25T19:28:58.362 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.360+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171786800 0 0) 0x7fad600aa1f0 con 0x7fad600b4700 2024-03-25T19:28:58.362 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.360+0000 7fad5dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.362 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.360+0000 7fad5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.362 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.360+0000 7fad5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.362 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.360+0000 7fad5dffb640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:28:58.362 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.360+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad600a9d60 con 0x7fad600b4700 2024-03-25T19:28:58.365 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.363+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198901171 0 0) 0x7fad6013ff10 con 0x7fad60142a20 2024-03-25T19:28:58.365 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.363+0000 7fad5dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.365 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.363+0000 7fad5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.365 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.363+0000 7fad5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.365 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.363+0000 7fad5dffb640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:28:58.365 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.363+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad600aa1f0 con 0x7fad60142a20 2024-03-25T19:28:58.365 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1994182269 0 0) 0x7fad48003520 con 0x7fad600a96b0 2024-03-25T19:28:58.365 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 >> v1:172.21.15.19:6790/0 conn(0x7fad600b4700 legacy=0x7fad60142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.365 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 >> v1:172.21.15.188:6789/0 conn(0x7fad60142a20 legacy=0x7fad6013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 1 monclient: found mon.noname-a 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad600b4ae0 con 0x7fad600a96b0 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 10 monclient: _finish_auth 0 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fad58001a10 con 0x7fad600a96b0 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad653c8640 5 monclient: authenticate success, global_id 4134 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad653c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:58.366 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad5dffb640 10 monclient: dump: 2024-03-25T19:28:58.367 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.367 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.367 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.367 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.367 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.367 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.367 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.367 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.367 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.367 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.367 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.364+0000 7fad653c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:58.370 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad58001bf0 con 0x7fad600a96b0 2024-03-25T19:28:58.370 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad5dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:58.370 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad5dffb640 1 -- 172.21.15.188:0/3398428500 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fad58002000 con 0x7fad600a96b0 2024-03-25T19:28:58.370 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad653c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:58.370 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad653c8640 4 set_mon_vals no callback set 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad5dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad5dffb640 10 monclient: dump: 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.371 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad653c8640 10 monclient: shutdown 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad653c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad653c8640 1 -- 172.21.15.188:0/3398428500 >> v1:172.21.15.19:6789/0 conn(0x7fad600a96b0 legacy=0x7fad6013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad653c8640 1 -- 172.21.15.188:0/3398428500 shutdown_connections 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad653c8640 1 -- 172.21.15.188:0/3398428500 >> 172.21.15.188:0/3398428500 conn(0x7fad600668e0 msgr2=0x7fad60066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad653c8640 1 -- 172.21.15.188:0/3398428500 shutdown_connections 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.366+0000 7fad653c8640 1 -- 172.21.15.188:0/3398428500 wait complete. 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.367+0000 7fad653c8640 10 monclient: build_initial_monmap 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.367+0000 7fad653c8640 10 monclient: monmap: 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 0.000000 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:created 0.000000 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:58.372 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 1 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.367+0000 7fad653c8640 1 librados: starting msgr at 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.367+0000 7fad653c8640 1 librados: starting objecter 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 1 Processor -- start 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 1 -- start start 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 1 librados: setting wanted keys 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 1 librados: calling monclient init 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 10 monclient: init 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 10 monclient(hunting): picked mon.noname-c con 0x7fad600a96b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:58.373 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 10 monclient(hunting): picked mon.noname-b con 0x7fad600b4700 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:58.374 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 10 monclient(hunting): picked mon.noname-a con 0x7fad60142a20 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:58.374 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad60135de0 con 0x7fad60142a20 2024-03-25T19:28:58.374 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad60135ae0 con 0x7fad600b4700 2024-03-25T19:28:58.374 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad600ac070 con 0x7fad600a96b0 2024-03-25T19:28:58.374 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:58.374 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.368+0000 7fad653c8640 10 monclient(hunting): authenticate will time out at 5992.685369s 2024-03-25T19:28:58.374 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.369+0000 7fad5e7fc640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fad600b4700 0x7fad601daa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:55674/0 (socket says 172.21.15.188:55674) 2024-03-25T19:28:58.374 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.369+0000 7fad5e7fc640 1 -- 172.21.15.188:0/4057628297 learned_addr learned my addr 172.21.15.188:0/4057628297 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:58.374 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.369+0000 7fad5effd640 1 --1- 172.21.15.188:0/4057628297 >> v1:172.21.15.19:6790/0 conn(0x7fad600a96b0 0x7fad601353d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.188:57920/0 (socket says 172.21.15.188:57920) 2024-03-25T19:28:58.374 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.369+0000 7fad477fe640 10 client.?.objecter ms_handle_connect 0x7fad600b4700 2024-03-25T19:28:58.374 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.369+0000 7fad477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:58.375 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.369+0000 7fad477fe640 10 client.?.objecter ms_handle_connect 0x7fad600a96b0 2024-03-25T19:28:58.375 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.369+0000 7fad477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:58.375 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.369+0000 7fad477fe640 10 client.?.objecter ms_handle_connect 0x7fad60142a20 2024-03-25T19:28:58.375 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.369+0000 7fad477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:58.376 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.371+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794756588 0 0) 0x7fad60135ae0 con 0x7fad600b4700 2024-03-25T19:28:58.376 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.371+0000 7fad477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.376 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.371+0000 7fad477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.376 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.371+0000 7fad477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.376 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.371+0000 7fad477fe640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:28:58.376 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.371+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad4c004eb0 con 0x7fad600b4700 2024-03-25T19:28:58.376 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.371+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181150581 0 0) 0x7fad60135de0 con 0x7fad60142a20 2024-03-25T19:28:58.376 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.371+0000 7fad477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.371+0000 7fad477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.371+0000 7fad477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.371+0000 7fad477fe640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:28:58.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.371+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad60135ae0 con 0x7fad60142a20 2024-03-25T19:28:58.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.374+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203562725 0 0) 0x7fad600ac070 con 0x7fad600a96b0 2024-03-25T19:28:58.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.374+0000 7fad477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.374+0000 7fad477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.374+0000 7fad477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.374+0000 7fad477fe640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:28:58.377 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.374+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad60135de0 con 0x7fad600a96b0 2024-03-25T19:28:58.378 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:58.378+0000 7f68fe44b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:28:58.378 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.376+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 448222343 0 0) 0x7fad4c004eb0 con 0x7fad600b4700 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.376+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad600ac070 con 0x7fad600b4700 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.376+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fad54001910 con 0x7fad600b4700 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.376+0000 7fad477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.376+0000 7fad477fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.376+0000 7fad477fe640 10 monclient(hunting): dump: 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.379 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.376+0000 7fad477fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:58.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.376+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 736160799 0 0) 0x7fad60135ae0 con 0x7fad60142a20 2024-03-25T19:28:58.380 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.376+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad4c004eb0 con 0x7fad60142a20 2024-03-25T19:28:58.382 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fad58001f00 con 0x7fad60142a20 2024-03-25T19:28:58.382 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.382 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:58.382 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 10 monclient(hunting): dump: 2024-03-25T19:28:58.382 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.382 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.382 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.382 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.382 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.383 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.383 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.383 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.383 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.385 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.385 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2348472658 0 0) 0x7fad4c004eb0 con 0x7fad60142a20 2024-03-25T19:28:58.385 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 >> v1:172.21.15.19:6790/0 conn(0x7fad600a96b0 legacy=0x7fad601353d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.385 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 >> v1:172.21.15.188:6789/0 conn(0x7fad600b4700 legacy=0x7fad601daa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.385 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 10 monclient: _finish_hunting 0 2024-03-25T19:28:58.385 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 1 monclient: found mon.a 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad60131590 con 0x7fad60142a20 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 10 monclient: _finish_auth 0 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 10 monclient: _check_auth_tickets 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad653c8640 5 monclient: authenticate success, global_id 4137 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad653c8640 10 monclient: _renew_subs 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad653c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad653c8640 1 -- 172.21.15.188:0/4057628297 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad60131840 con 0x7fad60142a20 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad653c8640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad653c8640 10 monclient: _renew_subs 2024-03-25T19:28:58.386 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad653c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.387 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad653c8640 1 -- 172.21.15.188:0/4057628297 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad601db340 con 0x7fad60142a20 2024-03-25T19:28:58.387 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.380+0000 7fad653c8640 1 librados: init done 2024-03-25T19:28:58.387 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.381+0000 7fad653c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:58.387 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.381+0000 7fad653c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:58.387 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.381+0000 7fad653c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.387 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.381+0000 7fad653c8640 1 -- 172.21.15.188:0/4057628297 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad60135f90 con 0x7fad60142a20 2024-03-25T19:28:58.388 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad58002060 con 0x7fad60142a20 2024-03-25T19:28:58.388 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:58.388 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fad58003c20 con 0x7fad60142a20 2024-03-25T19:28:58.388 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.388 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:58.388 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad477fe640 10 monclient: dump: 2024-03-25T19:28:58.388 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.388 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.389 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.389 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.389 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.389 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.389 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.389 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.389 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.389 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.389 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fad58004110 con 0x7fad60142a20 2024-03-25T19:28:58.389 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad5d7fa640 4 set_mon_vals no callback set 2024-03-25T19:28:58.389 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 569485202 0 0) 0x7fad58005780 con 0x7fad60142a20 2024-03-25T19:28:58.389 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad477fe640 10 client.4137.objecter ms_dispatch 0x7fad60000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:28:58.390 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad477fe640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:28:58.390 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad477fe640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:28:58.390 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.384+0000 7fad477fe640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T19:28:58.420 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:58.419+0000 7fcf0f22b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:28:58.465 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:58.464+0000 7fcf0f22b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:28:58.484 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.483+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fad580053a0 con 0x7fad60142a20 2024-03-25T19:28:58.484 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.483+0000 7fad477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:58.484 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.483+0000 7fad477fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:58.513 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:58.514+0000 7f68fe44b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:28:58.561 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:58.562+0000 7f68fe44b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:28:58.573 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.572+0000 7fad653c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8f6cea59-df77-45e1-960e-96fcfe4f78b4", "id": 3}] 2024-03-25T19:28:58.573 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.572+0000 7fad653c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8f6cea59-df77-45e1-960e-96fcfe4f78b4", "id": 3}] 2024-03-25T19:28:58.573 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.572+0000 7fad653c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.573 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.572+0000 7fad653c8640 1 -- 172.21.15.188:0/4057628297 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8f6cea59-df77-45e1-960e-96fcfe4f78b4", "id": 3} v 0) v1 -- 0x7fad601db640 con 0x7fad60142a20 2024-03-25T19:28:58.593 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:58.592+0000 7fcf0f22b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:28:58.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.606+0000 7fad477fe640 1 -- 172.21.15.188:0/4057628297 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f6cea59-df77-45e1-960e-96fcfe4f78b4", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1157751557 0 525046306) 0x7fad58005990 con 0x7fad60142a20 2024-03-25T19:28:58.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.606+0000 7fad477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8f6cea59-df77-45e1-960e-96fcfe4f78b4", "id": 3}] 2024-03-25T19:28:58.607 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.606+0000 7fad477fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:58.608 INFO:teuthology.orchestra.run.smithi188.stdout:3 2024-03-25T19:28:58.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.607+0000 7fad653c8640 10 librados: watch_flush enter 2024-03-25T19:28:58.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.607+0000 7fad653c8640 10 librados: watch_flush exit 2024-03-25T19:28:58.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.607+0000 7fad653c8640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T19:28:58.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.607+0000 7fad653c8640 10 client.4137.objecter successfully canceled tick 2024-03-25T19:28:58.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.607+0000 7fad653c8640 10 monclient: shutdown 2024-03-25T19:28:58.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.607+0000 7fad653c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:58.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.607+0000 7fad653c8640 1 -- 172.21.15.188:0/4057628297 >> v1:172.21.15.19:6789/0 conn(0x7fad60142a20 legacy=0x7fad601e2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.607+0000 7fad653c8640 1 -- 172.21.15.188:0/4057628297 shutdown_connections 2024-03-25T19:28:58.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.607+0000 7fad653c8640 1 -- 172.21.15.188:0/4057628297 >> 172.21.15.188:0/4057628297 conn(0x7fad600668e0 msgr2=0x7fad600b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:58.609 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.607+0000 7fad653c8640 1 -- 172.21.15.188:0/4057628297 shutdown_connections 2024-03-25T19:28:58.610 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.607+0000 7fad653c8640 1 -- 172.21.15.188:0/4057628297 wait complete. 2024-03-25T19:28:58.610 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.607+0000 7fad653c8640 1 librados: shutdown 2024-03-25T19:28:58.622 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new e72f23f5-3b0d-4046-80d1-074992d80443 4 2024-03-25T19:28:58.683 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:58.684+0000 7f68fe44b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:28:58.694 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:58.693+0000 7fcf0f22b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:28:58.763 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.762+0000 7fcba2c7b640 10 monclient: get_monmap_and_config 2024-03-25T19:28:58.763 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.762+0000 7fcba2c7b640 10 monclient: build_initial_monmap 2024-03-25T19:28:58.763 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.762+0000 7fcba2c7b640 10 monclient: monmap: 2024-03-25T19:28:58.763 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:58.763 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.763 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:58.764451+0000 2024-03-25T19:28:58.763 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:58.764451+0000 2024-03-25T19:28:58.763 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:58.764 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.764 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:58.764 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:28:58.764 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:28:58.764 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.765 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 1 Processor -- start 2024-03-25T19:28:58.765 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 1 -- start start 2024-03-25T19:28:58.765 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 10 monclient: init 2024-03-25T19:28:58.765 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:58.765 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:58.765 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 10 monclient(hunting): picked mon.noname-a con 0x7fcb9c0b4700 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:58.765 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 10 monclient(hunting): picked mon.noname-c con 0x7fcb9c142a20 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:58.766 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 10 monclient(hunting): picked mon.noname-b con 0x7fcb9c0a96b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:58.766 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c0a9d60 con 0x7fcb9c0b4700 2024-03-25T19:28:58.766 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c13ff10 con 0x7fcb9c142a20 2024-03-25T19:28:58.766 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c0aa1f0 con 0x7fcb9c0a96b0 2024-03-25T19:28:58.766 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:58.766 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.763+0000 7fcba2c7b640 10 monclient(hunting): authenticate will time out at 5993.080651s 2024-03-25T19:28:58.766 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.764+0000 7fcb9bfff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fcb9c142a20 0x7fcb9c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:46432/0 (socket says 172.21.15.188:46432) 2024-03-25T19:28:58.766 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.764+0000 7fcb9bfff640 1 -- 172.21.15.188:0/3099660843 learned_addr learned my addr 172.21.15.188:0/3099660843 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:58.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.766+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753311875 0 0) 0x7fcb9c13ff10 con 0x7fcb9c142a20 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.766+0000 7fcb9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.766+0000 7fcb9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.766+0000 7fcb9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.766+0000 7fcb9b7fe640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.766+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb8c003520 con 0x7fcb9c142a20 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.766+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164780526 0 0) 0x7fcb9c0aa1f0 con 0x7fcb9c0a96b0 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.767+0000 7fcb9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.767+0000 7fcb9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.767+0000 7fcb9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.767+0000 7fcb9b7fe640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.767+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb9c13ff10 con 0x7fcb9c0a96b0 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.767+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611631057 0 0) 0x7fcb9c0a9d60 con 0x7fcb9c0b4700 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.767+0000 7fcb9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.767+0000 7fcb9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.767+0000 7fcb9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.769 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.767+0000 7fcb9b7fe640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:28:58.769 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.767+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb9c0aa1f0 con 0x7fcb9c0b4700 2024-03-25T19:28:58.770 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 622931639 0 0) 0x7fcb8c003520 con 0x7fcb9c142a20 2024-03-25T19:28:58.770 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 >> v1:172.21.15.19:6790/0 conn(0x7fcb9c0a96b0 legacy=0x7fcb9c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.770 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 >> v1:172.21.15.19:6789/0 conn(0x7fcb9c0b4700 legacy=0x7fcb9c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.770 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:28:58.770 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:28:58.770 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:58.770 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:28:58.770 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c0b4ae0 con 0x7fcb9c142a20 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fcb88001690 con 0x7fcb9c142a20 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcba2c7b640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcba2c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcb9b7fe640 10 monclient: dump: 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.771 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.772 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.772 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.772 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.769+0000 7fcba2c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:58.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.771+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb880017f0 con 0x7fcb9c142a20 2024-03-25T19:28:58.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.771+0000 7fcb9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:58.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.771+0000 7fcb9b7fe640 1 -- 172.21.15.188:0/3099660843 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fcb88001c00 con 0x7fcb9c142a20 2024-03-25T19:28:58.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.771+0000 7fcb9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.772 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.771+0000 7fcb9b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.771+0000 7fcb9b7fe640 10 monclient: dump: 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.771+0000 7fcba2c7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:58.773 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.771+0000 7fcba2c7b640 4 set_mon_vals no callback set 2024-03-25T19:28:58.775 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.772+0000 7fcba2c7b640 10 monclient: shutdown 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.772+0000 7fcba2c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.772+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3099660843 >> v1:172.21.15.188:6789/0 conn(0x7fcb9c142a20 legacy=0x7fcb9c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.772+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3099660843 shutdown_connections 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.772+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3099660843 >> 172.21.15.188:0/3099660843 conn(0x7fcb9c0668e0 msgr2=0x7fcb9c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.772+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3099660843 shutdown_connections 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.772+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3099660843 wait complete. 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.773+0000 7fcba2c7b640 10 monclient: build_initial_monmap 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.773+0000 7fcba2c7b640 10 monclient: monmap: 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 0.000000 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:created 0.000000 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:58.776 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 1 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.773+0000 7fcba2c7b640 1 librados: starting msgr at 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.773+0000 7fcba2c7b640 1 librados: starting objecter 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.773+0000 7fcba2c7b640 1 Processor -- start 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.773+0000 7fcba2c7b640 1 -- start start 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.773+0000 7fcba2c7b640 1 librados: setting wanted keys 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.773+0000 7fcba2c7b640 1 librados: calling monclient init 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.773+0000 7fcba2c7b640 10 monclient: init 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.774+0000 7fcba2c7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.774+0000 7fcba2c7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.774+0000 7fcba2c7b640 10 monclient(hunting): picked mon.noname-b con 0x7fcb9c0a96b0 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:58.777 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.774+0000 7fcba2c7b640 10 monclient(hunting): picked mon.noname-c con 0x7fcb9c0b4700 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.774+0000 7fcba2c7b640 10 monclient(hunting): picked mon.noname-a con 0x7fcb9c142a20 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.774+0000 7fcba2c7b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c14eca0 con 0x7fcb9c142a20 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.774+0000 7fcba2c7b640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c1d74d0 con 0x7fcb9c0a96b0 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.774+0000 7fcba2c7b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c1d8350 con 0x7fcb9c0b4700 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.774+0000 7fcba2c7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.774+0000 7fcba2c7b640 10 monclient(hunting): authenticate will time out at 5993.091247s 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.774+0000 7fcba0a17640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fcb9c0a96b0 0x7fcb9c1d7c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:46434/0 (socket says 172.21.15.188:46434) 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.774+0000 7fcba0a17640 1 -- 172.21.15.188:0/3658297102 learned_addr learned my addr 172.21.15.188:0/3658297102 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.775+0000 7fcb997fa640 10 client.?.objecter ms_handle_connect 0x7fcb9c0a96b0 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.775+0000 7fcb997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.775+0000 7fcb997fa640 10 client.?.objecter ms_handle_connect 0x7fcb9c0b4700 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.775+0000 7fcb997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.775+0000 7fcb997fa640 10 client.?.objecter ms_handle_connect 0x7fcb9c142a20 2024-03-25T19:28:58.778 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.775+0000 7fcb997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:58.779 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674046521 0 0) 0x7fcb9c1d8350 con 0x7fcb9c0b4700 2024-03-25T19:28:58.779 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb80004eb0 con 0x7fcb9c0b4700 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157535687 0 0) 0x7fcb9c13ff10 con 0x7fcb9c142a20 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb9c1d8350 con 0x7fcb9c142a20 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108021243 0 0) 0x7fcb9c1d74d0 con 0x7fcb9c0a96b0 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:58.780 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:28:58.781 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.777+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb9c13ff10 con 0x7fcb9c0a96b0 2024-03-25T19:28:58.782 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:58.783+0000 7f68fe44b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:28:58.783 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3338640434 0 0) 0x7fcb9c1d8350 con 0x7fcb9c142a20 2024-03-25T19:28:58.783 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb9c1d74d0 con 0x7fcb9c142a20 2024-03-25T19:28:58.783 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fcb940019b0 con 0x7fcb9c142a20 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 10 monclient(hunting): dump: 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:58.784 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2902124139 0 0) 0x7fcb80004eb0 con 0x7fcb9c0b4700 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb9c1d8350 con 0x7fcb9c0b4700 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fcb88001b00 con 0x7fcb9c0b4700 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.782+0000 7fcb997fa640 10 monclient(hunting): dump: 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.785 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.786 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.784+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 28927596 0 0) 0x7fcb9c13ff10 con 0x7fcb9c0a96b0 2024-03-25T19:28:58.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.784+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb80004eb0 con 0x7fcb9c0a96b0 2024-03-25T19:28:58.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.784+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fcb900019b0 con 0x7fcb9c0a96b0 2024-03-25T19:28:58.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.784+0000 7fcb997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.784+0000 7fcb997fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:58.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.784+0000 7fcb997fa640 10 monclient(hunting): dump: 2024-03-25T19:28:58.787 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.787 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.787 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.787 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1927305529 0 0) 0x7fcb9c1d74d0 con 0x7fcb9c142a20 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 >> v1:172.21.15.19:6790/0 conn(0x7fcb9c0b4700 legacy=0x7fcb9c0acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 >> v1:172.21.15.188:6789/0 conn(0x7fcb9c0a96b0 legacy=0x7fcb9c1d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcb997fa640 10 monclient: _finish_hunting 0 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcb997fa640 1 monclient: found mon.a 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcb997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcb997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c0abd90 con 0x7fcb9c142a20 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcb997fa640 10 monclient: _finish_auth 0 2024-03-25T19:28:58.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcb997fa640 10 monclient: _check_auth_tickets 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcb997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcba2c7b640 5 monclient: authenticate success, global_id 4143 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcba2c7b640 10 monclient: _renew_subs 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcba2c7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3658297102 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9c1d7720 con 0x7fcb9c142a20 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcba2c7b640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcba2c7b640 10 monclient: _renew_subs 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcba2c7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3658297102 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9c0ad240 con 0x7fcb9c142a20 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.785+0000 7fcba2c7b640 1 librados: init done 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.786+0000 7fcba2c7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.786+0000 7fcba2c7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.786+0000 7fcba2c7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.786+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3658297102 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9c1d7900 con 0x7fcb9c142a20 2024-03-25T19:28:58.859 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:58.859+0000 7fcf0f22b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:28:58.890 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.889+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb94001ec0 con 0x7fcb9c142a20 2024-03-25T19:28:58.890 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.889+0000 7fcb997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:58.890 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.889+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fcb94003b10 con 0x7fcb9c142a20 2024-03-25T19:28:58.890 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.889+0000 7fcb997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:58.890 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.889+0000 7fcb997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:58.890 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.889+0000 7fcb997fa640 10 monclient: dump: 2024-03-25T19:28:58.890 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:58.891 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:58.891 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.891 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:58.891 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:58.891 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:58.891 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:58.891 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:58.891 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:58.891 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:58.891 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.889+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcb94004000 con 0x7fcb9c142a20 2024-03-25T19:28:58.891 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.889+0000 7fcb9affd640 4 set_mon_vals no callback set 2024-03-25T19:28:58.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.890+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2719184747 0 0) 0x7fcb940055a0 con 0x7fcb9c142a20 2024-03-25T19:28:58.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.890+0000 7fcb997fa640 10 client.4143.objecter ms_dispatch 0x7fcb9c000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:28:58.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.890+0000 7fcb997fa640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:28:58.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.890+0000 7fcb997fa640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:28:58.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.890+0000 7fcb997fa640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T19:28:58.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.890+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcb94005090 con 0x7fcb9c142a20 2024-03-25T19:28:58.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.890+0000 7fcb997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:58.892 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.890+0000 7fcb997fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:58.903 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:58.902+0000 7fcf0f22b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:28:58.944 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:58.945+0000 7f68fe44b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:28:58.962 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.961+0000 7fcba2c7b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e72f23f5-3b0d-4046-80d1-074992d80443", "id": 4}] 2024-03-25T19:28:58.962 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.961+0000 7fcba2c7b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e72f23f5-3b0d-4046-80d1-074992d80443", "id": 4}] 2024-03-25T19:28:58.962 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.961+0000 7fcba2c7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:28:58.963 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.961+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3658297102 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e72f23f5-3b0d-4046-80d1-074992d80443", "id": 4} v 0) v1 -- 0x7fcb9c04f730 con 0x7fcb9c142a20 2024-03-25T19:28:58.989 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:58.990+0000 7f68fe44b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:28:58.993 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.992+0000 7fcb997fa640 1 -- 172.21.15.188:0/3658297102 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e72f23f5-3b0d-4046-80d1-074992d80443", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 322740516 0 1697012583) 0x7fcb940052b0 con 0x7fcb9c142a20 2024-03-25T19:28:58.993 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.992+0000 7fcb997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e72f23f5-3b0d-4046-80d1-074992d80443", "id": 4}] 2024-03-25T19:28:58.993 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.992+0000 7fcb997fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:58.993 INFO:teuthology.orchestra.run.smithi188.stdout:4 2024-03-25T19:28:58.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.993+0000 7fcba2c7b640 10 librados: watch_flush enter 2024-03-25T19:28:58.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.993+0000 7fcba2c7b640 10 librados: watch_flush exit 2024-03-25T19:28:58.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.993+0000 7fcba2c7b640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T19:28:58.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.993+0000 7fcba2c7b640 10 client.4143.objecter successfully canceled tick 2024-03-25T19:28:58.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.993+0000 7fcba2c7b640 10 monclient: shutdown 2024-03-25T19:28:58.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.993+0000 7fcba2c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:58.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.993+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3658297102 >> v1:172.21.15.19:6789/0 conn(0x7fcb9c142a20 legacy=0x7fcb9c1d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.993+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3658297102 shutdown_connections 2024-03-25T19:28:58.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.993+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3658297102 >> 172.21.15.188:0/3658297102 conn(0x7fcb9c0668e0 msgr2=0x7fcb9c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:58.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.993+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3658297102 shutdown_connections 2024-03-25T19:28:58.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.993+0000 7fcba2c7b640 1 -- 172.21.15.188:0/3658297102 wait complete. 2024-03-25T19:28:58.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:58.993+0000 7fcba2c7b640 1 librados: shutdown 2024-03-25T19:28:59.004 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new bcee913e-5189-4fcd-b37e-2a62542e6c3f 5 2024-03-25T19:28:59.075 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:59.074+0000 7fcf0f22b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:28:59.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.144+0000 7fe8249a0640 10 monclient: get_monmap_and_config 2024-03-25T19:28:59.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.144+0000 7fe8249a0640 10 monclient: build_initial_monmap 2024-03-25T19:28:59.145 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.144+0000 7fe8249a0640 10 monclient: monmap: 2024-03-25T19:28:59.145 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:59.145 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.145 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:59.146059+0000 2024-03-25T19:28:59.145 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:59.146059+0000 2024-03-25T19:28:59.145 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:59.145 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.145 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:59.145 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:28:59.146 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:28:59.146 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.147 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.144+0000 7fe8249a0640 1 Processor -- start 2024-03-25T19:28:59.147 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.145+0000 7fe8249a0640 1 -- start start 2024-03-25T19:28:59.147 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.145+0000 7fe8249a0640 10 monclient: init 2024-03-25T19:28:59.147 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.145+0000 7fe8249a0640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:59.147 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.146+0000 7fe8249a0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:59.147 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.146+0000 7fe8249a0640 10 monclient(hunting): picked mon.noname-c con 0x7fe8200b4c70 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:59.147 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.146+0000 7fe8249a0640 10 monclient(hunting): picked mon.noname-b con 0x7fe8200b4250 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:59.147 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.146+0000 7fe8249a0640 10 monclient(hunting): picked mon.noname-a con 0x7fe8200a9dc0 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:59.148 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.146+0000 7fe8249a0640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8200aa210 con 0x7fe8200a9dc0 2024-03-25T19:28:59.148 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.146+0000 7fe8249a0640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8200ab200 con 0x7fe8200b4c70 2024-03-25T19:28:59.148 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.146+0000 7fe8249a0640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8200b3d30 con 0x7fe8200b4250 2024-03-25T19:28:59.148 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.146+0000 7fe8249a0640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:59.148 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.146+0000 7fe8249a0640 10 monclient(hunting): authenticate will time out at 5993.462816s 2024-03-25T19:28:59.149 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.148+0000 7fe81f7fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fe8200b4c70 0x7fe8201455f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:46450/0 (socket says 172.21.15.188:46450) 2024-03-25T19:28:59.149 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.148+0000 7fe81f7fe640 1 -- 172.21.15.188:0/3614331205 learned_addr learned my addr 172.21.15.188:0/3614331205 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:59.152 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:28:59.153+0000 7f68fe44b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:28:59.168 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.168+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943027760 0 0) 0x7fe8200aa210 con 0x7fe8200a9dc0 2024-03-25T19:28:59.169 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.168+0000 7fe81e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.169 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.168+0000 7fe81e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.169 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.168+0000 7fe81e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.169 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.168+0000 7fe81e7fc640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:28:59.169 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.168+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe808003520 con 0x7fe8200a9dc0 2024-03-25T19:28:59.169 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.168+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372055781 0 0) 0x7fe8200ab200 con 0x7fe8200b4c70 2024-03-25T19:28:59.169 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.168+0000 7fe81e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.168+0000 7fe81e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.168+0000 7fe81e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.168+0000 7fe81e7fc640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:28:59.170 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.168+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8200aa210 con 0x7fe8200b4c70 2024-03-25T19:28:59.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.170+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662088984 0 0) 0x7fe8200b3d30 con 0x7fe8200b4250 2024-03-25T19:28:59.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.170+0000 7fe81e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.170+0000 7fe81e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.170+0000 7fe81e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.170+0000 7fe81e7fc640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:28:59.171 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.170+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8200ab200 con 0x7fe8200b4250 2024-03-25T19:28:59.173 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 592319210 0 0) 0x7fe8200aa210 con 0x7fe8200b4c70 2024-03-25T19:28:59.173 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 >> v1:172.21.15.19:6790/0 conn(0x7fe8200b4250 legacy=0x7fe82013f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.173 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 >> v1:172.21.15.19:6789/0 conn(0x7fe8200a9dc0 legacy=0x7fe82014d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.173 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:59.173 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:28:59.173 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:59.173 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8200b2230 con 0x7fe8200b4c70 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe814001670 con 0x7fe8200b4c70 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe8249a0640 5 monclient: authenticate success, global_id 4135 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe8249a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe81e7fc640 10 monclient: dump: 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.174 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.175 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.175 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.175 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:59.175 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:59.175 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:59.175 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.175 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.172+0000 7fe8249a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:59.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8140017d0 con 0x7fe8200b4c70 2024-03-25T19:28:59.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe81e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:59.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe81e7fc640 1 -- 172.21.15.188:0/3614331205 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe814001be0 con 0x7fe8200b4c70 2024-03-25T19:28:59.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe8249a0640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:59.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe8249a0640 4 set_mon_vals no callback set 2024-03-25T19:28:59.176 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe81e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe81e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe81e7fc640 10 monclient: dump: 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe8249a0640 10 monclient: shutdown 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe8249a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:59.177 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe8249a0640 1 -- 172.21.15.188:0/3614331205 >> v1:172.21.15.188:6789/0 conn(0x7fe8200b4c70 legacy=0x7fe8201455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe8249a0640 1 -- 172.21.15.188:0/3614331205 shutdown_connections 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.174+0000 7fe8249a0640 1 -- 172.21.15.188:0/3614331205 >> 172.21.15.188:0/3614331205 conn(0x7fe820066ac0 msgr2=0x7fe820066ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.175+0000 7fe8249a0640 1 -- 172.21.15.188:0/3614331205 shutdown_connections 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.175+0000 7fe8249a0640 1 -- 172.21.15.188:0/3614331205 wait complete. 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.175+0000 7fe8249a0640 10 monclient: build_initial_monmap 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.176+0000 7fe8249a0640 10 monclient: monmap: 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 0.000000 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:created 0.000000 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 1 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:59.178 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:28:59.179 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:28:59.179 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.176+0000 7fe8249a0640 1 librados: starting msgr at 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.176+0000 7fe8249a0640 1 librados: starting objecter 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.176+0000 7fe8249a0640 1 Processor -- start 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.176+0000 7fe8249a0640 1 -- start start 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.176+0000 7fe8249a0640 1 librados: setting wanted keys 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.176+0000 7fe8249a0640 1 librados: calling monclient init 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.176+0000 7fe8249a0640 10 monclient: init 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.177+0000 7fe8249a0640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.177+0000 7fe8249a0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.177+0000 7fe8249a0640 10 monclient(hunting): picked mon.noname-b con 0x7fe8200a9dc0 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.177+0000 7fe8249a0640 10 monclient(hunting): picked mon.noname-c con 0x7fe8200b4250 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.177+0000 7fe8249a0640 10 monclient(hunting): picked mon.noname-a con 0x7fe8200b4c70 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.177+0000 7fe8249a0640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe820142750 con 0x7fe8200b4c70 2024-03-25T19:28:59.181 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.177+0000 7fe8249a0640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8201429c0 con 0x7fe8200a9dc0 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.177+0000 7fe8249a0640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82013fe90 con 0x7fe8200b4250 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.177+0000 7fe8249a0640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.177+0000 7fe8249a0640 10 monclient(hunting): authenticate will time out at 5993.494480s 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.178+0000 7fe81f7fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fe8200a9dc0 0x7fe820135ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:46464/0 (socket says 172.21.15.188:46464) 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.178+0000 7fe81f7fe640 1 -- 172.21.15.188:0/1253921157 learned_addr learned my addr 172.21.15.188:0/1253921157 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.178+0000 7fe807fff640 10 client.?.objecter ms_handle_connect 0x7fe8200a9dc0 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.178+0000 7fe807fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.178+0000 7fe807fff640 10 client.?.objecter ms_handle_connect 0x7fe8200b4250 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.178+0000 7fe807fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.180+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440258258 0 0) 0x7fe8201429c0 con 0x7fe8200a9dc0 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.180+0000 7fe807fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.180+0000 7fe807fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.180+0000 7fe807fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.180+0000 7fe807fff640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:28:59.182 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.180+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe80c004eb0 con 0x7fe8200a9dc0 2024-03-25T19:28:59.183 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.181+0000 7fe807fff640 10 client.?.objecter ms_handle_connect 0x7fe8200b4c70 2024-03-25T19:28:59.183 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.181+0000 7fe807fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:59.183 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.182+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531829405 0 0) 0x7fe82013fe90 con 0x7fe8200b4250 2024-03-25T19:28:59.183 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.182+0000 7fe807fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.183 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.182+0000 7fe807fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.184 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.182+0000 7fe807fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.184 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.182+0000 7fe807fff640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:28:59.184 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.182+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8201429c0 con 0x7fe8200b4250 2024-03-25T19:28:59.184 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.183+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967021958 0 0) 0x7fe820142750 con 0x7fe8200b4c70 2024-03-25T19:28:59.184 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.183+0000 7fe807fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.184 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.183+0000 7fe807fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.184 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.183+0000 7fe807fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.184 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.183+0000 7fe807fff640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:28:59.184 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.183+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe82013fe90 con 0x7fe8200b4c70 2024-03-25T19:28:59.186 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.185+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 660388510 0 0) 0x7fe80c004eb0 con 0x7fe8200a9dc0 2024-03-25T19:28:59.186 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.185+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe820142750 con 0x7fe8200a9dc0 2024-03-25T19:28:59.186 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.185+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe814001c20 con 0x7fe8200a9dc0 2024-03-25T19:28:59.186 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.185+0000 7fe807fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.186 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.185+0000 7fe807fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.185+0000 7fe807fff640 10 monclient(hunting): dump: 2024-03-25T19:28:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:59.187 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.187 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.185+0000 7fe807fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:59.191 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2756494911 0 0) 0x7fe82013fe90 con 0x7fe8200b4c70 2024-03-25T19:28:59.191 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe80c004eb0 con 0x7fe8200b4c70 2024-03-25T19:28:59.191 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe818001a30 con 0x7fe8200b4c70 2024-03-25T19:28:59.191 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.191 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:59.191 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 10 monclient(hunting): dump: 2024-03-25T19:28:59.191 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:59.191 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.191 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.191 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 357479938 0 0) 0x7fe8201429c0 con 0x7fe8200b4250 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe82013fe90 con 0x7fe8200b4250 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe810001870 con 0x7fe8200b4250 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 10 monclient(hunting): dump: 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.192 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.193 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.193 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.193 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:59.193 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:59.193 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:59.193 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3488860389 0 0) 0x7fe820142750 con 0x7fe8200a9dc0 2024-03-25T19:28:59.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 >> v1:172.21.15.19:6790/0 conn(0x7fe8200b4250 legacy=0x7fe8201d73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 >> v1:172.21.15.19:6789/0 conn(0x7fe8200b4c70 legacy=0x7fe8201e2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:59.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 1 monclient: found mon.b 2024-03-25T19:28:59.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.188+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8201405d0 con 0x7fe8200a9dc0 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.189+0000 7fe807fff640 10 monclient: _finish_auth 0 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.189+0000 7fe807fff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.189+0000 7fe807fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.189+0000 7fe8249a0640 5 monclient: authenticate success, global_id 4138 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.189+0000 7fe8249a0640 10 monclient: _renew_subs 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.189+0000 7fe8249a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.189+0000 7fe8249a0640 1 -- 172.21.15.188:0/1253921157 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe820140880 con 0x7fe8200a9dc0 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.189+0000 7fe8249a0640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.189+0000 7fe8249a0640 10 monclient: _renew_subs 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.189+0000 7fe8249a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.189+0000 7fe8249a0640 1 -- 172.21.15.188:0/1253921157 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8201365d0 con 0x7fe8200a9dc0 2024-03-25T19:28:59.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.189+0000 7fe8249a0640 1 librados: init done 2024-03-25T19:28:59.195 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.190+0000 7fe8249a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:59.195 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.190+0000 7fe8249a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:59.195 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.190+0000 7fe8249a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:28:59.195 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.190+0000 7fe8249a0640 1 -- 172.21.15.188:0/1253921157 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8200627b0 con 0x7fe8200a9dc0 2024-03-25T19:28:59.195 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.193+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe814004a90 con 0x7fe8200a9dc0 2024-03-25T19:28:59.195 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.193+0000 7fe807fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:59.195 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.193+0000 7fe81dffb640 4 set_mon_vals no callback set 2024-03-25T19:28:59.383 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe814004de0 con 0x7fe8200a9dc0 2024-03-25T19:28:59.383 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 10 monclient: dump: 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe814005350 con 0x7fe8200a9dc0 2024-03-25T19:28:59.384 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3711181591 0 0) 0x7fe8140065d0 con 0x7fe8200a9dc0 2024-03-25T19:28:59.385 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 10 client.4138.objecter ms_dispatch 0x7fe820000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:28:59.385 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 3 client.4138.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:28:59.385 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 3 client.4138.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:28:59.385 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T19:28:59.385 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe814018e00 con 0x7fe8200a9dc0 2024-03-25T19:28:59.385 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:59.385 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.382+0000 7fe807fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:59.454 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.453+0000 7fe8249a0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bcee913e-5189-4fcd-b37e-2a62542e6c3f", "id": 5}] 2024-03-25T19:28:59.454 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.453+0000 7fe8249a0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bcee913e-5189-4fcd-b37e-2a62542e6c3f", "id": 5}] 2024-03-25T19:28:59.454 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.453+0000 7fe8249a0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:28:59.454 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.453+0000 7fe8249a0640 1 -- 172.21.15.188:0/1253921157 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bcee913e-5189-4fcd-b37e-2a62542e6c3f", "id": 5} v 0) v1 -- 0x7fe820075010 con 0x7fe8200a9dc0 2024-03-25T19:28:59.587 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.586+0000 7fe807fff640 1 -- 172.21.15.188:0/1253921157 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bcee913e-5189-4fcd-b37e-2a62542e6c3f", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3955624335 0 1988412176) 0x7fe814006910 con 0x7fe8200a9dc0 2024-03-25T19:28:59.587 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.586+0000 7fe807fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bcee913e-5189-4fcd-b37e-2a62542e6c3f", "id": 5}] 2024-03-25T19:28:59.587 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.586+0000 7fe807fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:59.587 INFO:teuthology.orchestra.run.smithi188.stdout:5 2024-03-25T19:28:59.588 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.586+0000 7fe8249a0640 10 librados: watch_flush enter 2024-03-25T19:28:59.588 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.586+0000 7fe8249a0640 10 librados: watch_flush exit 2024-03-25T19:28:59.588 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.586+0000 7fe8249a0640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T19:28:59.588 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.586+0000 7fe8249a0640 10 client.4138.objecter successfully canceled tick 2024-03-25T19:28:59.589 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.587+0000 7fe8249a0640 10 monclient: shutdown 2024-03-25T19:28:59.589 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.587+0000 7fe8249a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:59.589 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.587+0000 7fe8249a0640 1 -- 172.21.15.188:0/1253921157 >> v1:172.21.15.188:6789/0 conn(0x7fe8200a9dc0 legacy=0x7fe820135ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.589 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.587+0000 7fe8249a0640 1 -- 172.21.15.188:0/1253921157 shutdown_connections 2024-03-25T19:28:59.589 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.587+0000 7fe8249a0640 1 -- 172.21.15.188:0/1253921157 >> 172.21.15.188:0/1253921157 conn(0x7fe820066ac0 msgr2=0x7fe8200b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:59.589 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.587+0000 7fe8249a0640 1 -- 172.21.15.188:0/1253921157 shutdown_connections 2024-03-25T19:28:59.589 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.587+0000 7fe8249a0640 1 -- 172.21.15.188:0/1253921157 wait complete. 2024-03-25T19:28:59.589 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.587+0000 7fe8249a0640 1 librados: shutdown 2024-03-25T19:28:59.599 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new eb7504bd-4506-4bdb-b58a-260851ce16b5 6 2024-03-25T19:28:59.742 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.741+0000 7f82bf7b6640 10 monclient: get_monmap_and_config 2024-03-25T19:28:59.742 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.741+0000 7f82bf7b6640 10 monclient: build_initial_monmap 2024-03-25T19:28:59.742 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.741+0000 7f82bf7b6640 10 monclient: monmap: 2024-03-25T19:28:59.742 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:59.742 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.742 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:59.743476+0000 2024-03-25T19:28:59.742 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:59.743476+0000 2024-03-25T19:28:59.743 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:59.743 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.743 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:59.743 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:28:59.743 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:28:59.743 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.744 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 1 Processor -- start 2024-03-25T19:28:59.744 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 1 -- start start 2024-03-25T19:28:59.744 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 10 monclient: init 2024-03-25T19:28:59.744 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:59.744 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:59.744 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 10 monclient(hunting): picked mon.noname-b con 0x7f82b80b4700 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:59.744 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 10 monclient(hunting): picked mon.noname-c con 0x7f82b8142a20 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:59.745 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 10 monclient(hunting): picked mon.noname-a con 0x7f82b80a96b0 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:59.745 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b80a9d60 con 0x7f82b80a96b0 2024-03-25T19:28:59.745 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b813ff10 con 0x7f82b8142a20 2024-03-25T19:28:59.745 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b80aa1f0 con 0x7f82b80b4700 2024-03-25T19:28:59.745 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:59.745 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.742+0000 7f82bf7b6640 10 monclient(hunting): authenticate will time out at 5994.059635s 2024-03-25T19:28:59.745 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.743+0000 7f82bcd51640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f82b8142a20 0x7f82b813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.188:46480/0 (socket says 172.21.15.188:46480) 2024-03-25T19:28:59.745 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.743+0000 7f82bcd51640 1 -- 172.21.15.188:0/405107324 learned_addr learned my addr 172.21.15.188:0/405107324 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:59.746 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478071515 0 0) 0x7f82b80a9d60 con 0x7f82b80a96b0 2024-03-25T19:28:59.746 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.746 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.747 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.747 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:28:59.747 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a4003520 con 0x7f82b80a96b0 2024-03-25T19:28:59.747 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597074082 0 0) 0x7f82b813ff10 con 0x7f82b8142a20 2024-03-25T19:28:59.747 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.748 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.748 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.748 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:28:59.748 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b80a9d60 con 0x7f82b8142a20 2024-03-25T19:28:59.748 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159480726 0 0) 0x7f82b80aa1f0 con 0x7f82b80b4700 2024-03-25T19:28:59.748 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.748 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.748 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.748 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:28:59.748 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.746+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b813ff10 con 0x7f82b80b4700 2024-03-25T19:28:59.750 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2691249495 0 0) 0x7f82a4003520 con 0x7f82b80a96b0 2024-03-25T19:28:59.750 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 >> v1:172.21.15.19:6790/0 conn(0x7f82b80b4700 legacy=0x7f82b8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.750 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 >> v1:172.21.15.188:6789/0 conn(0x7f82b8142a20 legacy=0x7f82b813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.750 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:59.750 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 1 monclient: found mon.noname-a 2024-03-25T19:28:59.750 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:59.750 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:28:59.750 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b80b4ae0 con 0x7f82b80a96b0 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 10 monclient: _finish_auth 0 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f82b0001a10 con 0x7f82b80a96b0 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82affff640 10 monclient: dump: 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:59.751 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.752 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82bf7b6640 5 monclient: authenticate success, global_id 4152 2024-03-25T19:28:59.752 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.749+0000 7f82bf7b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:59.753 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82b0001b70 con 0x7f82b80a96b0 2024-03-25T19:28:59.753 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:59.753 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82affff640 1 -- 172.21.15.188:0/405107324 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f82b0001f80 con 0x7f82b80a96b0 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82affff640 10 monclient: dump: 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82bf7b6640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82bf7b6640 4 set_mon_vals no callback set 2024-03-25T19:28:59.754 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82bf7b6640 10 monclient: shutdown 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82bf7b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82bf7b6640 1 -- 172.21.15.188:0/405107324 >> v1:172.21.15.19:6789/0 conn(0x7f82b80a96b0 legacy=0x7f82b813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82bf7b6640 1 -- 172.21.15.188:0/405107324 shutdown_connections 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82bf7b6640 1 -- 172.21.15.188:0/405107324 >> 172.21.15.188:0/405107324 conn(0x7f82b80668e0 msgr2=0x7f82b8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82bf7b6640 1 -- 172.21.15.188:0/405107324 shutdown_connections 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.751+0000 7f82bf7b6640 1 -- 172.21.15.188:0/405107324 wait complete. 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.752+0000 7f82bf7b6640 10 monclient: build_initial_monmap 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.752+0000 7f82bf7b6640 10 monclient: monmap: 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 0.000000 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:created 0.000000 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 1 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:28:59.755 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:28:59.756 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:28:59.756 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.756 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.752+0000 7f82bf7b6640 1 librados: starting msgr at 2024-03-25T19:28:59.756 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.752+0000 7f82bf7b6640 1 librados: starting objecter 2024-03-25T19:28:59.756 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.752+0000 7f82bf7b6640 1 Processor -- start 2024-03-25T19:28:59.756 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 1 -- start start 2024-03-25T19:28:59.756 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 1 librados: setting wanted keys 2024-03-25T19:28:59.756 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 1 librados: calling monclient init 2024-03-25T19:28:59.756 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 10 monclient: init 2024-03-25T19:28:59.758 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:59.758 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:59.758 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 10 monclient(hunting): picked mon.noname-b con 0x7f82b80a96b0 addr v1:172.21.15.188:6789/0 2024-03-25T19:28:59.758 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 10 monclient(hunting): picked mon.noname-a con 0x7f82b80b4700 addr v1:172.21.15.19:6789/0 2024-03-25T19:28:59.758 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 10 monclient(hunting): picked mon.noname-c con 0x7f82b8142a20 addr v1:172.21.15.19:6790/0 2024-03-25T19:28:59.758 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b80aba20 con 0x7f82b80b4700 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b80abd00 con 0x7f82b80a96b0 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b813bd80 con 0x7f82b8142a20 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.753+0000 7f82bf7b6640 10 monclient(hunting): authenticate will time out at 5994.070292s 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.754+0000 7f82bdd53640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f82b8142a20 0x7f82b81e2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.188:59472/0 (socket says 172.21.15.188:59472) 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.754+0000 7f82bdd53640 1 -- 172.21.15.188:0/588710781 learned_addr learned my addr 172.21.15.188:0/588710781 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.754+0000 7f82adffb640 10 client.?.objecter ms_handle_connect 0x7f82b8142a20 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.754+0000 7f82adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.754+0000 7f82adffb640 10 client.?.objecter ms_handle_connect 0x7f82b80a96b0 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.754+0000 7f82adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.754+0000 7f82adffb640 10 client.?.objecter ms_handle_connect 0x7f82b80b4700 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.754+0000 7f82adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681099559 0 0) 0x7f82b80abd00 con 0x7f82b80a96b0 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.759 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c004eb0 con 0x7f82b80a96b0 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310309534 0 0) 0x7f82b80aba20 con 0x7f82b80b4700 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b80abd00 con 0x7f82b80b4700 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999952212 0 0) 0x7f82b813bd80 con 0x7f82b8142a20 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:28:59.760 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.756+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b80aba20 con 0x7f82b8142a20 2024-03-25T19:28:59.761 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.760+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 823971405 0 0) 0x7f829c004eb0 con 0x7f82b80a96b0 2024-03-25T19:28:59.761 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.760+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82b813bd80 con 0x7f82b80a96b0 2024-03-25T19:28:59.761 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.760+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f82b40018f0 con 0x7f82b80a96b0 2024-03-25T19:28:59.761 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.760+0000 7f82adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.761 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.760+0000 7f82adffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:59.762 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.760+0000 7f82adffb640 10 monclient(hunting): dump: 2024-03-25T19:28:59.762 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:59.762 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.762 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.762 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.762 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.762 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.762 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:59.762 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:59.762 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:59.762 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.762 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.760+0000 7f82adffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.762+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2236482438 0 0) 0x7f82b80aba20 con 0x7f82b8142a20 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.762+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f829c004eb0 con 0x7f82b8142a20 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.762+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f82b0001e80 con 0x7f82b8142a20 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.762+0000 7f82adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.762+0000 7f82adffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.762+0000 7f82adffb640 10 monclient(hunting): dump: 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.763 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:59.764 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:59.764 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:59.764 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.764 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.763+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3898249194 0 0) 0x7f82b80abd00 con 0x7f82b80b4700 2024-03-25T19:28:59.764 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.763+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82b80aba20 con 0x7f82b80b4700 2024-03-25T19:28:59.764 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.763+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f82a8001970 con 0x7f82b80b4700 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.763+0000 7f82adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.763+0000 7f82adffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.763+0000 7f82adffb640 10 monclient(hunting): dump: 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:59.765 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.766 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1937145964 0 0) 0x7f829c004eb0 con 0x7f82b8142a20 2024-03-25T19:28:59.766 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 >> v1:172.21.15.188:6789/0 conn(0x7f82b80a96b0 legacy=0x7f82b80ac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 >> v1:172.21.15.19:6789/0 conn(0x7f82b80b4700 legacy=0x7f82b81392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82adffb640 10 monclient: _finish_hunting 0 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82adffb640 1 monclient: found mon.c 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b813ccb0 con 0x7f82b8142a20 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82adffb640 10 monclient: _finish_auth 0 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82adffb640 10 monclient: _check_auth_tickets 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82bf7b6640 5 monclient: authenticate success, global_id 4142 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82bf7b6640 10 monclient: _renew_subs 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82bf7b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82bf7b6640 1 -- 172.21.15.188:0/588710781 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b813ac10 con 0x7f82b8142a20 2024-03-25T19:28:59.767 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82bf7b6640 10 client.4142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:59.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82bf7b6640 10 monclient: _renew_subs 2024-03-25T19:28:59.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82bf7b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:28:59.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82bf7b6640 1 -- 172.21.15.188:0/588710781 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b813b8d0 con 0x7f82b8142a20 2024-03-25T19:28:59.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.765+0000 7f82bf7b6640 1 librados: init done 2024-03-25T19:28:59.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.766+0000 7f82bf7b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:59.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.766+0000 7f82bf7b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:59.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.766+0000 7f82bf7b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:28:59.768 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.766+0000 7f82bf7b6640 1 -- 172.21.15.188:0/588710781 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b813bf10 con 0x7f82b8142a20 2024-03-25T19:28:59.937 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:28:59.936+0000 7fcf0f22b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:28:59.994 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.993+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82b0001fe0 con 0x7f82b8142a20 2024-03-25T19:28:59.994 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.993+0000 7f82adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:59.994 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.993+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f82b00015d0 con 0x7f82b8142a20 2024-03-25T19:28:59.994 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.993+0000 7f82adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:59.994 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.993+0000 7f82adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:59.994 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.993+0000 7f82adffb640 10 monclient: dump: 2024-03-25T19:28:59.994 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:28:59.994 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:28:59.994 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.994 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:28:59.994 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:59.994 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:28:59.995 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:28:59.995 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:28:59.995 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:28:59.995 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:28:59.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.993+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f82b0004210 con 0x7f82b8142a20 2024-03-25T19:28:59.995 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.993+0000 7f82af7fe640 4 set_mon_vals no callback set 2024-03-25T19:28:59.996 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.995+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 658827546 0 0) 0x7f82b0006ac0 con 0x7f82b8142a20 2024-03-25T19:28:59.997 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.995+0000 7f82adffb640 10 client.4142.objecter ms_dispatch 0x7f82b8000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:28:59.997 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.995+0000 7f82adffb640 3 client.4142.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:28:59.997 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.995+0000 7f82adffb640 3 client.4142.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:28:59.997 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.995+0000 7f82adffb640 20 client.4142.objecter dump_active .. 0 homeless 2024-03-25T19:28:59.997 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.996+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f82b00046f0 con 0x7f82b8142a20 2024-03-25T19:28:59.997 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.996+0000 7f82adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:59.997 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:28:59.996+0000 7f82adffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:00.022 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:29:00.021+0000 7f68fe44b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:29:00.069 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.068+0000 7f82bf7b6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eb7504bd-4506-4bdb-b58a-260851ce16b5", "id": 6}] 2024-03-25T19:29:00.069 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.068+0000 7f82bf7b6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eb7504bd-4506-4bdb-b58a-260851ce16b5", "id": 6}] 2024-03-25T19:29:00.069 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.068+0000 7f82bf7b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:29:00.069 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.068+0000 7f82bf7b6640 1 -- 172.21.15.188:0/588710781 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd new", "uuid": "eb7504bd-4506-4bdb-b58a-260851ce16b5", "id": 6} v 0) v1 -- 0x7f82b813bb80 con 0x7f82b8142a20 2024-03-25T19:29:00.188 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.187+0000 7f82adffb640 1 -- 172.21.15.188:0/588710781 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eb7504bd-4506-4bdb-b58a-260851ce16b5", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3588147899 0 1113811849) 0x7f82b00064d0 con 0x7f82b8142a20 2024-03-25T19:29:00.188 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.187+0000 7f82adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eb7504bd-4506-4bdb-b58a-260851ce16b5", "id": 6}] 2024-03-25T19:29:00.188 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.187+0000 7f82adffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:00.188 INFO:teuthology.orchestra.run.smithi188.stdout:6 2024-03-25T19:29:00.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.188+0000 7f82bf7b6640 10 librados: watch_flush enter 2024-03-25T19:29:00.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.188+0000 7f82bf7b6640 10 librados: watch_flush exit 2024-03-25T19:29:00.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.188+0000 7f82bf7b6640 20 client.4142.objecter shutdown clearing up homeless session... 2024-03-25T19:29:00.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.188+0000 7f82bf7b6640 10 client.4142.objecter successfully canceled tick 2024-03-25T19:29:00.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.188+0000 7f82bf7b6640 10 monclient: shutdown 2024-03-25T19:29:00.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.188+0000 7f82bf7b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:00.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.188+0000 7f82bf7b6640 1 -- 172.21.15.188:0/588710781 >> v1:172.21.15.19:6790/0 conn(0x7f82b8142a20 legacy=0x7f82b81e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:00.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.188+0000 7f82bf7b6640 1 -- 172.21.15.188:0/588710781 shutdown_connections 2024-03-25T19:29:00.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.188+0000 7f82bf7b6640 1 -- 172.21.15.188:0/588710781 >> 172.21.15.188:0/588710781 conn(0x7f82b80668e0 msgr2=0x7f82b813d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:00.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.188+0000 7f82bf7b6640 1 -- 172.21.15.188:0/588710781 shutdown_connections 2024-03-25T19:29:00.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.188+0000 7f82bf7b6640 1 -- 172.21.15.188:0/588710781 wait complete. 2024-03-25T19:29:00.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.188+0000 7f82bf7b6640 1 librados: shutdown 2024-03-25T19:29:00.199 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster ceph osd new 3e4577eb-444c-4860-afd9-f52b1b2f00d3 7 2024-03-25T19:29:00.216 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:29:00.215+0000 7fcf0f22b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:29:00.281 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:29:00.280+0000 7fcf0f22b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:29:00.300 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.299+0000 7f1bc0fef640 10 monclient: get_monmap_and_config 2024-03-25T19:29:00.300 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.299+0000 7f1bc0fef640 10 monclient: build_initial_monmap 2024-03-25T19:29:00.300 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.299+0000 7f1bc0fef640 10 monclient: monmap: 2024-03-25T19:29:00.300 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:29:00.301 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:00.301 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:29:00.301218+0000 2024-03-25T19:29:00.301 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:29:00.301218+0000 2024-03-25T19:29:00.301 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:00.301 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:29:00.301 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:00.301 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:29:00.301 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:29:00.301 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:29:00.301 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.299+0000 7f1bc0fef640 1 Processor -- start 2024-03-25T19:29:00.301 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.299+0000 7f1bc0fef640 1 -- start start 2024-03-25T19:29:00.301 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.299+0000 7f1bc0fef640 10 monclient: init 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.300+0000 7f1bc0fef640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.300+0000 7f1bc0fef640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.300+0000 7f1bc0fef640 10 monclient(hunting): picked mon.noname-a con 0x7f1bbc0b4700 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.300+0000 7f1bc0fef640 10 monclient(hunting): picked mon.noname-b con 0x7f1bbc142a20 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.300+0000 7f1bc0fef640 10 monclient(hunting): picked mon.noname-c con 0x7f1bbc0a96b0 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.300+0000 7f1bc0fef640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bbc0a9d60 con 0x7f1bbc0b4700 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.300+0000 7f1bc0fef640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bbc13ff10 con 0x7f1bbc0a96b0 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.300+0000 7f1bc0fef640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bbc0aa1f0 con 0x7f1bbc142a20 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.300+0000 7f1bc0fef640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.300+0000 7f1bc0fef640 10 monclient(hunting): authenticate will time out at 5994.617412s 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.301+0000 7f1bba59c640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f1bbc0b4700 0x7f1bbc142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.188:50478/0 (socket says 172.21.15.188:50478) 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.301+0000 7f1bb9d9b640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f1bbc142a20 0x7f1bbc13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.188:59482/0 (socket says 172.21.15.188:59482) 2024-03-25T19:29:00.303 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.301+0000 7f1bba59c640 1 -- 172.21.15.188:0/729406979 learned_addr learned my addr 172.21.15.188:0/729406979 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:29:00.304 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.303+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458958608 0 0) 0x7f1bbc0a9d60 con 0x7f1bbc0b4700 2024-03-25T19:29:00.304 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.303+0000 7f1bb959a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.303+0000 7f1bb959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.303+0000 7f1bb959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.303+0000 7f1bb959a640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.303+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba4003520 con 0x7f1bbc0b4700 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.303+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163870437 0 0) 0x7f1bbc0aa1f0 con 0x7f1bbc142a20 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.303+0000 7f1bb959a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.303+0000 7f1bb959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.303+0000 7f1bb959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.303+0000 7f1bb959a640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.303+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bbc0a9d60 con 0x7f1bbc142a20 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.304+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835807257 0 0) 0x7f1bbc13ff10 con 0x7f1bbc0a96b0 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.304+0000 7f1bb959a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.304+0000 7f1bb959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:00.305 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.304+0000 7f1bb959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:00.306 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.304+0000 7f1bb959a640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:29:00.306 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.304+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bbc0aa1f0 con 0x7f1bbc0a96b0 2024-03-25T19:29:00.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1169217570 0 0) 0x7f1ba4003520 con 0x7f1bbc0b4700 2024-03-25T19:29:00.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 >> v1:172.21.15.19:6790/0 conn(0x7f1bbc142a20 legacy=0x7f1bbc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:00.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 >> v1:172.21.15.188:6789/0 conn(0x7f1bbc0a96b0 legacy=0x7f1bbc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:00.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 10 monclient: _finish_hunting 0 2024-03-25T19:29:00.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 1 monclient: found mon.noname-a 2024-03-25T19:29:00.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:00.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bbc0b4ae0 con 0x7f1bbc0b4700 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 10 monclient: _finish_auth 0 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 10 monclient: _check_auth_tickets 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bc0fef640 5 monclient: authenticate success, global_id 4158 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1bb0001690 con 0x7f1bbc0b4700 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bc0fef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bb959a640 10 monclient: dump: 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:00.309 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:00.310 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:29:00.310 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:00.310 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:00.310 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:00.310 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:29:00.310 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.307+0000 7f1bc0fef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:00.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.309+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bb00017f0 con 0x7f1bbc0b4700 2024-03-25T19:29:00.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.309+0000 7f1bb959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:00.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.309+0000 7f1bb959a640 1 -- 172.21.15.188:0/729406979 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1bb0001c00 con 0x7f1bbc0b4700 2024-03-25T19:29:00.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.309+0000 7f1bc0fef640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:00.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.309+0000 7f1bc0fef640 4 set_mon_vals no callback set 2024-03-25T19:29:00.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.309+0000 7f1bb959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:00.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.309+0000 7f1bb959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:00.311 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.309+0000 7f1bb959a640 10 monclient: dump: 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.310+0000 7f1bc0fef640 10 monclient: shutdown 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.310+0000 7f1bc0fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.310+0000 7f1bc0fef640 1 -- 172.21.15.188:0/729406979 >> v1:172.21.15.19:6789/0 conn(0x7f1bbc0b4700 legacy=0x7f1bbc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.310+0000 7f1bc0fef640 1 -- 172.21.15.188:0/729406979 shutdown_connections 2024-03-25T19:29:00.312 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.310+0000 7f1bc0fef640 1 -- 172.21.15.188:0/729406979 >> 172.21.15.188:0/729406979 conn(0x7f1bbc0668e0 msgr2=0x7f1bbc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:00.313 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.310+0000 7f1bc0fef640 1 -- 172.21.15.188:0/729406979 shutdown_connections 2024-03-25T19:29:00.313 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.310+0000 7f1bc0fef640 1 -- 172.21.15.188:0/729406979 wait complete. 2024-03-25T19:29:00.315 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.311+0000 7f1bc0fef640 10 monclient: build_initial_monmap 2024-03-25T19:29:00.315 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.311+0000 7f1bc0fef640 10 monclient: monmap: 2024-03-25T19:29:00.315 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 0 2024-03-25T19:29:00.315 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:00.315 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 0.000000 2024-03-25T19:29:00.315 INFO:teuthology.orchestra.run.smithi188.stderr:created 0.000000 2024-03-25T19:29:00.315 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:00.315 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 1 2024-03-25T19:29:00.315 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:00.315 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.311+0000 7f1bc0fef640 1 librados: starting msgr at 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.311+0000 7f1bc0fef640 1 librados: starting objecter 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.311+0000 7f1bc0fef640 1 Processor -- start 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.311+0000 7f1bc0fef640 1 -- start start 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.311+0000 7f1bc0fef640 1 librados: setting wanted keys 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.311+0000 7f1bc0fef640 1 librados: calling monclient init 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.311+0000 7f1bc0fef640 10 monclient: init 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.312+0000 7f1bc0fef640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.312+0000 7f1bc0fef640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.312+0000 7f1bc0fef640 10 monclient(hunting): picked mon.noname-c con 0x7f1bbc0a96b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.312+0000 7f1bc0fef640 10 monclient(hunting): picked mon.noname-a con 0x7f1bbc0b4700 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.312+0000 7f1bc0fef640 10 monclient(hunting): picked mon.noname-b con 0x7f1bbc142a20 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:00.316 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.312+0000 7f1bc0fef640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bbc0abf00 con 0x7f1bbc0b4700 2024-03-25T19:29:00.317 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.312+0000 7f1bc0fef640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bbc1d4fe0 con 0x7f1bbc142a20 2024-03-25T19:29:00.317 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.312+0000 7f1bc0fef640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bbc0ac450 con 0x7f1bbc0a96b0 2024-03-25T19:29:00.317 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.312+0000 7f1bc0fef640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:00.317 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.312+0000 7f1bc0fef640 10 monclient(hunting): authenticate will time out at 5994.628928s 2024-03-25T19:29:00.317 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.312+0000 7f1bba59c640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f1bbc0a96b0 0x7f1bbc1dabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.188:59492/0 (socket says 172.21.15.188:59492) 2024-03-25T19:29:00.317 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.312+0000 7f1bba59c640 1 -- 172.21.15.188:0/1814556913 learned_addr learned my addr 172.21.15.188:0/1814556913 (peer_addr_for_me v1:172.21.15.188:0/0) 2024-03-25T19:29:00.317 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.313+0000 7f1baaffd640 10 client.?.objecter ms_handle_connect 0x7f1bbc0a96b0 2024-03-25T19:29:00.317 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.313+0000 7f1baaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:00.317 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.313+0000 7f1baaffd640 10 client.?.objecter ms_handle_connect 0x7f1bbc142a20 2024-03-25T19:29:00.317 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.313+0000 7f1baaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:00.317 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.314+0000 7f1baaffd640 10 client.?.objecter ms_handle_connect 0x7f1bbc0b4700 2024-03-25T19:29:00.317 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.314+0000 7f1baaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:00.318 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:29:00.316+0000 7f68fe44b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.315+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812517067 0 0) 0x7f1bbc1d4fe0 con 0x7f1bbc142a20 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.315+0000 7f1baaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.315+0000 7f1baaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.315+0000 7f1baaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.315+0000 7f1baaffd640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.315+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b9c004eb0 con 0x7f1bbc142a20 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.315+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359023406 0 0) 0x7f1bbc0ac450 con 0x7f1bbc0a96b0 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.315+0000 7f1baaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.315+0000 7f1baaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.315+0000 7f1baaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.315+0000 7f1baaffd640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.315+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bbc1d4fe0 con 0x7f1bbc0a96b0 2024-03-25T19:29:00.319 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.316+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915183319 0 0) 0x7f1bbc0abf00 con 0x7f1bbc0b4700 2024-03-25T19:29:00.320 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.316+0000 7f1baaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:00.320 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.316+0000 7f1baaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:00.320 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.316+0000 7f1baaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:00.320 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.316+0000 7f1baaffd640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:29:00.320 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.316+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bbc0ac450 con 0x7f1bbc0b4700 2024-03-25T19:29:00.321 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 562473910 0 0) 0x7f1b9c004eb0 con 0x7f1bbc142a20 2024-03-25T19:29:00.321 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bbc0abf00 con 0x7f1bbc142a20 2024-03-25T19:29:00.321 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1bb4001bd0 con 0x7f1bbc142a20 2024-03-25T19:29:00.321 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:00.321 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:00.321 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 10 monclient(hunting): dump: 2024-03-25T19:29:00.321 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:29:00.321 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 802156583 0 0) 0x7f1bbc0ac450 con 0x7f1bbc0b4700 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b9c004eb0 con 0x7f1bbc0b4700 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1bac0018f0 con 0x7f1bbc0b4700 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:00.322 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 10 monclient(hunting): dump: 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 381365116 0 0) 0x7f1bbc1d4fe0 con 0x7f1bbc0a96b0 2024-03-25T19:29:00.323 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.320+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bbc0ac450 con 0x7f1bbc0a96b0 2024-03-25T19:29:00.324 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2262282526 0 0) 0x7f1bbc0abf00 con 0x7f1bbc142a20 2024-03-25T19:29:00.324 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 >> v1:172.21.15.19:6790/0 conn(0x7f1bbc0a96b0 legacy=0x7f1bbc1dabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:00.324 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 >> v1:172.21.15.19:6789/0 conn(0x7f1bbc0b4700 legacy=0x7f1bbc1db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1baaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1baaffd640 1 monclient: found mon.b 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1baaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1baaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bbc0ac5e0 con 0x7f1bbc142a20 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1baaffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1baaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1baaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1bc0fef640 5 monclient: authenticate success, global_id 4150 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1bc0fef640 10 monclient: _renew_subs 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1bc0fef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1bc0fef640 1 -- 172.21.15.188:0/1814556913 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bbc0ab880 con 0x7f1bbc142a20 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1bc0fef640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:00.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1bc0fef640 10 monclient: _renew_subs 2024-03-25T19:29:00.326 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1bc0fef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:00.326 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1bc0fef640 1 -- 172.21.15.188:0/1814556913 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bbc1e33b0 con 0x7f1bbc142a20 2024-03-25T19:29:00.326 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.323+0000 7f1bc0fef640 1 librados: init done 2024-03-25T19:29:00.326 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.324+0000 7f1bc0fef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:00.326 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.324+0000 7f1bc0fef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:00.326 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.324+0000 7f1bc0fef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:00.326 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.324+0000 7f1bc0fef640 1 -- 172.21.15.188:0/1814556913 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bbc0ac7d0 con 0x7f1bbc142a20 2024-03-25T19:29:00.329 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bb4002060 con 0x7f1bbc142a20 2024-03-25T19:29:00.329 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1baaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:00.329 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1bb4003b30 con 0x7f1bbc142a20 2024-03-25T19:29:00.329 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1baaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:00.329 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1baaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:00.329 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1baaffd640 10 monclient: dump: 2024-03-25T19:29:00.329 INFO:teuthology.orchestra.run.smithi188.stderr:epoch 1 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:election_strategy: 3 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1bb4004060 con 0x7f1bbc142a20 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1bb8d99640 4 set_mon_vals no callback set 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2943376057 0 0) 0x7f1bb40058c0 con 0x7f1bbc142a20 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1baaffd640 10 client.4150.objecter ms_dispatch 0x7f1bbc000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:29:00.330 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1baaffd640 3 client.4150.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:29:00.331 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1baaffd640 3 client.4150.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:29:00.331 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.328+0000 7f1baaffd640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T19:29:00.369 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:29:00.370+0000 7f68fe44b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:29:00.404 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:29:00.403+0000 7fcf0f22b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:29:00.421 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.420+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1bb4005ad0 con 0x7f1bbc142a20 2024-03-25T19:29:00.421 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.420+0000 7f1baaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:00.421 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.420+0000 7f1baaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:00.490 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-25T19:29:00.491+0000 7f68fe44b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:29:00.492 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.490+0000 7f1bc0fef640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3e4577eb-444c-4860-afd9-f52b1b2f00d3", "id": 7}] 2024-03-25T19:29:00.492 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.490+0000 7f1bc0fef640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3e4577eb-444c-4860-afd9-f52b1b2f00d3", "id": 7}] 2024-03-25T19:29:00.492 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.490+0000 7f1bc0fef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:00.492 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.490+0000 7f1bc0fef640 1 -- 172.21.15.188:0/1814556913 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3e4577eb-444c-4860-afd9-f52b1b2f00d3", "id": 7} v 0) v1 -- 0x7f1bbc0adbc0 con 0x7f1bbc142a20 2024-03-25T19:29:00.706 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.705+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 349141752 0 0) 0x7f1bb4005370 con 0x7f1bbc142a20 2024-03-25T19:29:00.917 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.917+0000 7f1baaffd640 1 -- 172.21.15.188:0/1814556913 <== mon.1 v1:172.21.15.188:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3e4577eb-444c-4860-afd9-f52b1b2f00d3", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3626125410 0 1371665406) 0x7f1bb4014e90 con 0x7f1bbc142a20 2024-03-25T19:29:00.918 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.917+0000 7f1baaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3e4577eb-444c-4860-afd9-f52b1b2f00d3", "id": 7}] 2024-03-25T19:29:00.918 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.917+0000 7f1baaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:00.918 INFO:teuthology.orchestra.run.smithi188.stdout:7 2024-03-25T19:29:00.919 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.918+0000 7f1bc0fef640 10 librados: watch_flush enter 2024-03-25T19:29:00.919 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.918+0000 7f1bc0fef640 10 librados: watch_flush exit 2024-03-25T19:29:00.919 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.918+0000 7f1bc0fef640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T19:29:00.919 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.918+0000 7f1bc0fef640 10 client.4150.objecter successfully canceled tick 2024-03-25T19:29:00.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.918+0000 7f1bc0fef640 10 monclient: shutdown 2024-03-25T19:29:00.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.918+0000 7f1bc0fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:00.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.918+0000 7f1bc0fef640 1 -- 172.21.15.188:0/1814556913 >> v1:172.21.15.188:6789/0 conn(0x7f1bbc142a20 legacy=0x7f1bbc1e1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:00.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.918+0000 7f1bc0fef640 1 -- 172.21.15.188:0/1814556913 shutdown_connections 2024-03-25T19:29:00.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.918+0000 7f1bc0fef640 1 -- 172.21.15.188:0/1814556913 >> 172.21.15.188:0/1814556913 conn(0x7f1bbc0668e0 msgr2=0x7f1bbc0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:00.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.918+0000 7f1bc0fef640 1 -- 172.21.15.188:0/1814556913 shutdown_connections 2024-03-25T19:29:00.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.918+0000 7f1bc0fef640 1 -- 172.21.15.188:0/1814556913 wait complete. 2024-03-25T19:29:00.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-03-25T19:29:00.918+0000 7f1bc0fef640 1 librados: shutdown 2024-03-25T19:29:00.929 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:29:00.929 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:29:00.929 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:29:00.931 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:29:00.932 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:29:00.932 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:29:00.932 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:29:00.934 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:29:00.934 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:29:00.934 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:29:00.934 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:29:00.936 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:29:00.936 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:29:00.937 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:29:00.937 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:29:00.939 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:29:00.939 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:29:00.939 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:29:00.939 DEBUG:teuthology.orchestra.run.smithi188:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:29:00.972 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:29:00.972 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:29:00.972 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:29:00.972 DEBUG:teuthology.orchestra.run.smithi188:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:29:00.974 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:29:00.974 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:29:00.974 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:29:00.974 DEBUG:teuthology.orchestra.run.smithi188:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:29:00.976 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:29:00.976 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:29:00.976 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:29:00.976 DEBUG:teuthology.orchestra.run.smithi188:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:29:00.978 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:29:00.979 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:01.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.144+0000 7f2641655640 10 monclient: get_monmap_and_config 2024-03-25T19:29:01.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.144+0000 7f2641655640 10 monclient: build_initial_monmap 2024-03-25T19:29:01.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.144+0000 7f2641655640 10 monclient: monmap: 2024-03-25T19:29:01.143 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-03-25T19:29:01.143 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:01.143 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-25T19:29:01.144408+0000 2024-03-25T19:29:01.143 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-25T19:29:01.144408+0000 2024-03-25T19:29:01.144 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:01.144 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:01.144 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:01.144 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:29:01.144 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:29:01.144 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:29:01.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.145+0000 7f2641655640 1 Processor -- start 2024-03-25T19:29:01.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.145+0000 7f2641655640 1 -- start start 2024-03-25T19:29:01.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.145+0000 7f2641655640 10 monclient: init 2024-03-25T19:29:01.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.146+0000 7f2641655640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:01.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.146+0000 7f2641655640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:01.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.146+0000 7f2641655640 10 monclient(hunting): picked mon.noname-b con 0x7f263c0b4700 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:01.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.146+0000 7f2641655640 10 monclient(hunting): picked mon.noname-a con 0x7f263c142a20 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:01.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.146+0000 7f2641655640 10 monclient(hunting): picked mon.noname-c con 0x7f263c0a96b0 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:01.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.146+0000 7f2641655640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c0a9d60 con 0x7f263c142a20 2024-03-25T19:29:01.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.146+0000 7f2641655640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c13ff10 con 0x7f263c0a96b0 2024-03-25T19:29:01.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.146+0000 7f2641655640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c0aa1f0 con 0x7f263c0b4700 2024-03-25T19:29:01.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.146+0000 7f2641655640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:01.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.146+0000 7f2641655640 10 monclient(hunting): authenticate will time out at 5994.415081s 2024-03-25T19:29:01.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.147+0000 7f263b7fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f263c0a96b0 0x7f263c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:37728/0 (socket says 172.21.15.19:37728) 2024-03-25T19:29:01.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.147+0000 7f263b7fe640 1 -- 172.21.15.19:0/472024017 learned_addr learned my addr 172.21.15.19:0/472024017 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:01.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.281+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148540899 0 0) 0x7f263c0a9d60 con 0x7f263c142a20 2024-03-25T19:29:01.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.282+0000 7f2639ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:01.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.282+0000 7f2639ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:01.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.282+0000 7f2639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:01.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.282+0000 7f2639ffb640 10 monclient(hunting): my global_id is 4170 2024-03-25T19:29:01.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.282+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2624003520 con 0x7f263c142a20 2024-03-25T19:29:01.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.290+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367549934 0 0) 0x7f263c13ff10 con 0x7f263c0a96b0 2024-03-25T19:29:01.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.290+0000 7f2639ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:01.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.290+0000 7f2639ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:01.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.290+0000 7f2639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:01.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.290+0000 7f2639ffb640 10 monclient(hunting): my global_id is 4168 2024-03-25T19:29:01.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.290+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f263c0a9d60 con 0x7f263c0a96b0 2024-03-25T19:29:01.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.292+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187688633 0 0) 0x7f263c0aa1f0 con 0x7f263c0b4700 2024-03-25T19:29:01.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.292+0000 7f2639ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:01.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.292+0000 7f2639ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:01.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.292+0000 7f2639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:01.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.292+0000 7f2639ffb640 10 monclient(hunting): my global_id is 4160 2024-03-25T19:29:01.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.292+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f263c13ff10 con 0x7f263c0b4700 2024-03-25T19:29:01.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 19529122 0 0) 0x7f2624003520 con 0x7f263c142a20 2024-03-25T19:29:01.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 >> v1:172.21.15.19:6790/0 conn(0x7f263c0b4700 legacy=0x7f263c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:01.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 >> v1:172.21.15.188:6789/0 conn(0x7f263c0a96b0 legacy=0x7f263c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:01.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:29:01.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 1 monclient: found mon.noname-a 2024-03-25T19:29:01.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:01.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:29:01.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c0b4ae0 con 0x7f263c142a20 2024-03-25T19:29:01.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 10 monclient: _finish_auth 0 2024-03-25T19:29:01.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:29:01.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:01.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f262c001690 con 0x7f263c142a20 2024-03-25T19:29:01.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2641655640 5 monclient: authenticate success, global_id 4170 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2641655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2639ffb640 10 monclient: dump: 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:29:01.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.312+0000 7f2641655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:01.503 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.503+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f262c0017f0 con 0x7f263c142a20 2024-03-25T19:29:01.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.503+0000 7f2639ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:01.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.503+0000 7f2639ffb640 1 -- 172.21.15.19:0/472024017 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f262c001c00 con 0x7f263c142a20 2024-03-25T19:29:01.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.503+0000 7f2639ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:01.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.503+0000 7f2639ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:01.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.503+0000 7f2639ffb640 10 monclient: dump: 2024-03-25T19:29:01.504 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-25T19:29:01.504 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:01.504 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:01.504 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:01.505 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:01.505 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:01.505 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:01.505 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:01.505 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:01.505 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:29:01.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.503+0000 7f2641655640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:01.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.503+0000 7f2641655640 4 set_mon_vals no callback set 2024-03-25T19:29:01.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.504+0000 7f2641655640 10 monclient: shutdown 2024-03-25T19:29:01.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.504+0000 7f2641655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:01.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.504+0000 7f2641655640 1 -- 172.21.15.19:0/472024017 >> v1:172.21.15.19:6789/0 conn(0x7f263c142a20 legacy=0x7f263c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:01.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.504+0000 7f2641655640 1 -- 172.21.15.19:0/472024017 shutdown_connections 2024-03-25T19:29:01.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.504+0000 7f2641655640 1 -- 172.21.15.19:0/472024017 >> 172.21.15.19:0/472024017 conn(0x7f263c0668e0 msgr2=0x7f263c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:01.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.504+0000 7f2641655640 1 -- 172.21.15.19:0/472024017 shutdown_connections 2024-03-25T19:29:01.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.504+0000 7f2641655640 1 -- 172.21.15.19:0/472024017 wait complete. 2024-03-25T19:29:01.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.505+0000 7f2641655640 10 monclient: build_initial_monmap 2024-03-25T19:29:01.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.505+0000 7f2641655640 10 monclient: monmap: 2024-03-25T19:29:01.510 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-03-25T19:29:01.510 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:01.510 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-03-25T19:29:01.510 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-03-25T19:29:01.510 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:01.510 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-25T19:29:01.510 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.506+0000 7f2641655640 1 librados: starting msgr at 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.506+0000 7f2641655640 1 librados: starting objecter 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.506+0000 7f2641655640 1 Processor -- start 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.506+0000 7f2641655640 1 -- start start 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.506+0000 7f2641655640 1 librados: setting wanted keys 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.506+0000 7f2641655640 1 librados: calling monclient init 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.506+0000 7f2641655640 10 monclient: init 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.507+0000 7f2641655640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.507+0000 7f2641655640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.507+0000 7f2641655640 10 monclient(hunting): picked mon.noname-b con 0x7f263c0a96b0 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.507+0000 7f2641655640 10 monclient(hunting): picked mon.noname-c con 0x7f263c0b4700 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:01.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.507+0000 7f2641655640 10 monclient(hunting): picked mon.noname-a con 0x7f263c142a20 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:01.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.507+0000 7f2641655640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c06df50 con 0x7f263c142a20 2024-03-25T19:29:01.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.507+0000 7f2641655640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c06fa00 con 0x7f263c0a96b0 2024-03-25T19:29:01.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.507+0000 7f2641655640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c06fce0 con 0x7f263c0b4700 2024-03-25T19:29:01.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.507+0000 7f2641655640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:01.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.507+0000 7f2641655640 10 monclient(hunting): authenticate will time out at 5994.777388s 2024-03-25T19:29:01.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.508+0000 7f263affd640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f263c0a96b0 0x7f263c06e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:37730/0 (socket says 172.21.15.19:37730) 2024-03-25T19:29:01.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.508+0000 7f263affd640 1 -- 172.21.15.19:0/1917757928 learned_addr learned my addr 172.21.15.19:0/1917757928 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:01.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.509+0000 7f26237fe640 10 client.?.objecter ms_handle_connect 0x7f263c0a96b0 2024-03-25T19:29:01.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.509+0000 7f26237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:01.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.510+0000 7f26237fe640 10 client.?.objecter ms_handle_connect 0x7f263c0b4700 2024-03-25T19:29:01.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.510+0000 7f26237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:01.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.548+0000 7f26237fe640 10 client.?.objecter ms_handle_connect 0x7f263c142a20 2024-03-25T19:29:01.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.548+0000 7f26237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:01.610 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.610+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766945745 0 0) 0x7f263c06df50 con 0x7f263c142a20 2024-03-25T19:29:01.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.610+0000 7f26237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:01.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.610+0000 7f26237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:01.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.610+0000 7f26237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:01.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.610+0000 7f26237fe640 10 monclient(hunting): my global_id is 4173 2024-03-25T19:29:01.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.610+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2628004eb0 con 0x7f263c142a20 2024-03-25T19:29:01.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.659+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2734426113 0 0) 0x7f2628004eb0 con 0x7f263c142a20 2024-03-25T19:29:01.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.659+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f263c06df50 con 0x7f263c142a20 2024-03-25T19:29:01.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.662+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2634001cf0 con 0x7f263c142a20 2024-03-25T19:29:01.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.662+0000 7f26237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:01.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.662+0000 7f26237fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:01.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.662+0000 7f26237fe640 10 monclient(hunting): dump: 2024-03-25T19:29:01.662 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-25T19:29:01.662 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:01.662 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:01.662 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:01.662 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:01.662 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:01.662 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:01.662 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:01.662 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:01.662 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:29:01.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.662+0000 7f26237fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:01.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.673+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4192964457 0 0) 0x7f263c06df50 con 0x7f263c142a20 2024-03-25T19:29:01.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 >> v1:172.21.15.19:6790/0 conn(0x7f263c0b4700 legacy=0x7f263c14be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:01.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 >> v1:172.21.15.188:6789/0 conn(0x7f263c0a96b0 legacy=0x7f263c06e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:01.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f26237fe640 10 monclient: _finish_hunting 0 2024-03-25T19:29:01.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f26237fe640 1 monclient: found mon.a 2024-03-25T19:29:01.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f26237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:01.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f26237fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:01.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c06ffc0 con 0x7f263c142a20 2024-03-25T19:29:01.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f26237fe640 10 monclient: _finish_auth 0 2024-03-25T19:29:01.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f26237fe640 10 monclient: _check_auth_tickets 2024-03-25T19:29:01.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f26237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:01.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f2641655640 5 monclient: authenticate success, global_id 4173 2024-03-25T19:29:01.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f2641655640 10 monclient: _renew_subs 2024-03-25T19:29:01.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f2641655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:01.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f2641655640 1 -- 172.21.15.19:0/1917757928 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c14e5d0 con 0x7f263c142a20 2024-03-25T19:29:01.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f2641655640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:01.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f2641655640 10 monclient: _renew_subs 2024-03-25T19:29:01.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f2641655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:01.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f2641655640 1 -- 172.21.15.19:0/1917757928 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f263c1e74c0 con 0x7f263c142a20 2024-03-25T19:29:01.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.674+0000 7f2641655640 1 librados: init done 2024-03-25T19:29:01.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.675+0000 7f2641655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:01.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.675+0000 7f2641655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:01.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.675+0000 7f2641655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:01.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.675+0000 7f2641655640 1 -- 172.21.15.19:0/1917757928 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263c04f730 con 0x7f263c142a20 2024-03-25T19:29:01.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.678+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2634004b60 con 0x7f263c142a20 2024-03-25T19:29:01.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.678+0000 7f26237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:01.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.678+0000 7f26397fa640 4 set_mon_vals no callback set 2024-03-25T19:29:01.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.679+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2634004e70 con 0x7f263c142a20 2024-03-25T19:29:01.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.679+0000 7f26237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:01.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.679+0000 7f26237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:01.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.679+0000 7f26237fe640 10 monclient: dump: 2024-03-25T19:29:01.679 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-25T19:29:01.679 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:01.679 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:01.679 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:01.679 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:01.680 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:01.680 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:01.680 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:01.680 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:01.680 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:29:01.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.861+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 349141752 0 0) 0x7f263400ff50 con 0x7f263c142a20 2024-03-25T19:29:01.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.871+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f2634015ae0 con 0x7f263c142a20 2024-03-25T19:29:01.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.871+0000 7f26237fe640 10 client.4173.objecter ms_dispatch 0x7f263c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:01.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.871+0000 7f26237fe640 3 client.4173.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:01.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.871+0000 7f26237fe640 3 client.4173.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:01.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.871+0000 7f26237fe640 20 client.4173.objecter dump_active .. 0 homeless 2024-03-25T19:29:01.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.871+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2634014da0 con 0x7f263c142a20 2024-03-25T19:29:01.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.871+0000 7f26237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:01.872 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:01.871+0000 7f26237fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:02.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.000+0000 7f2641655640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:02.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.000+0000 7f2641655640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:02.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.000+0000 7f2641655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:02.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.000+0000 7f2641655640 1 -- 172.21.15.19:0/1917757928 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f263c069f30 con 0x7f263c142a20 2024-03-25T19:29:02.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.252+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 475583741 0 0) 0x7f2634011ad0 con 0x7f263c142a20 2024-03-25T19:29:02.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.253+0000 7f263affd640 10 monclient: get_auth_request con 0x7f262806e190 auth_method 0 2024-03-25T19:29:02.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.253+0000 7f26237fe640 10 client.4173.objecter ms_handle_connect 0x7f262806e190 2024-03-25T19:29:02.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.398+0000 7f26237fe640 1 -- 172.21.15.19:0/1917757928 <== mon.0 v1:172.21.15.19:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f2634021130 con 0x7f263c142a20 2024-03-25T19:29:02.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.398+0000 7f26237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:02.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.398+0000 7f26237fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:02.399 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-25T19:29:02.399 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:02.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.399+0000 7f2641655640 10 librados: watch_flush enter 2024-03-25T19:29:02.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.399+0000 7f2641655640 10 librados: watch_flush exit 2024-03-25T19:29:02.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.399+0000 7f2641655640 20 client.4173.objecter shutdown clearing up homeless session... 2024-03-25T19:29:02.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.399+0000 7f2641655640 10 client.4173.objecter successfully canceled tick 2024-03-25T19:29:02.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.399+0000 7f2641655640 1 -- 172.21.15.19:0/1917757928 >> v1:172.21.15.188:6800/2207506229 conn(0x7f262806e190 legacy=0x7f2628070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.400+0000 7f2641655640 10 monclient: shutdown 2024-03-25T19:29:02.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.400+0000 7f2641655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:02.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.400+0000 7f2641655640 1 -- 172.21.15.19:0/1917757928 >> v1:172.21.15.19:6789/0 conn(0x7f263c142a20 legacy=0x7f263c14c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.400+0000 7f2641655640 1 -- 172.21.15.19:0/1917757928 shutdown_connections 2024-03-25T19:29:02.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.400+0000 7f2641655640 1 -- 172.21.15.19:0/1917757928 >> 172.21.15.19:0/1917757928 conn(0x7f263c0668e0 msgr2=0x7f263c0b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:02.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.400+0000 7f2641655640 1 -- 172.21.15.19:0/1917757928 shutdown_connections 2024-03-25T19:29:02.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.400+0000 7f2641655640 1 -- 172.21.15.19:0/1917757928 wait complete. 2024-03-25T19:29:02.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-25T19:29:02.400+0000 7f2641655640 1 librados: shutdown 2024-03-25T19:29:02.416 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:29:02.416 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:29:02.540 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:29:02.539+0000 7fceaf810640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:29:02.546 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:29:02.546+0000 7fceaf810640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:29:03.222 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:03.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.387+0000 7f1d33582640 10 monclient: get_monmap_and_config 2024-03-25T19:29:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.387+0000 7f1d33582640 10 monclient: build_initial_monmap 2024-03-25T19:29:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.387+0000 7f1d33582640 10 monclient: monmap: 2024-03-25T19:29:03.388 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:03.388 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:03.388 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:29:03.388738+0000 2024-03-25T19:29:03.388 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:29:03.388738+0000 2024-03-25T19:29:03.388 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:03.388 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:03.388 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:03.388 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:29:03.388 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:29:03.388 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.388+0000 7f1d33582640 1 Processor -- start 2024-03-25T19:29:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.388+0000 7f1d33582640 1 -- start start 2024-03-25T19:29:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.388+0000 7f1d33582640 10 monclient: init 2024-03-25T19:29:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.389+0000 7f1d33582640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.389+0000 7f1d33582640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.389+0000 7f1d33582640 10 monclient(hunting): picked mon.noname-a con 0x7f1d2c0b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.389+0000 7f1d33582640 10 monclient(hunting): picked mon.noname-b con 0x7f1d2c0b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.389+0000 7f1d33582640 10 monclient(hunting): picked mon.noname-c con 0x7f1d2c14be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.389+0000 7f1d33582640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c0af940 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.389+0000 7f1d33582640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c0b0a70 con 0x7f1d2c14be90 2024-03-25T19:29:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.389+0000 7f1d33582640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c0b0d50 con 0x7f1d2c0b1730 2024-03-25T19:29:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.389+0000 7f1d33582640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:03.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.389+0000 7f1d33582640 10 monclient(hunting): authenticate will time out at 5996.659685s 2024-03-25T19:29:03.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.390+0000 7f1d30b1d640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f1d2c0b1730 0x7f1d2c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:42052/0 (socket says 172.21.15.19:42052) 2024-03-25T19:29:03.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.390+0000 7f1d30b1d640 1 -- 172.21.15.19:0/4228259708 learned_addr learned my addr 172.21.15.19:0/4228259708 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:03.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.394+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030223212 0 0) 0x7f1d2c0b0d50 con 0x7f1d2c0b1730 2024-03-25T19:29:03.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.394+0000 7f1d23fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:03.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.394+0000 7f1d23fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:03.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.394+0000 7f1d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:03.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.394+0000 7f1d23fff640 10 monclient(hunting): my global_id is 4163 2024-03-25T19:29:03.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.394+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d18003520 con 0x7f1d2c0b1730 2024-03-25T19:29:03.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.394+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894653365 0 0) 0x7f1d2c0af940 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.394+0000 7f1d23fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:03.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.394+0000 7f1d23fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:03.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.394+0000 7f1d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:03.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.394+0000 7f1d23fff640 10 monclient(hunting): my global_id is 4176 2024-03-25T19:29:03.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.394+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c0b0d50 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.395+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552153388 0 0) 0x7f1d2c0b0a70 con 0x7f1d2c14be90 2024-03-25T19:29:03.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.395+0000 7f1d23fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:03.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.395+0000 7f1d23fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:03.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.395+0000 7f1d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:03.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.395+0000 7f1d23fff640 10 monclient(hunting): my global_id is 4174 2024-03-25T19:29:03.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.395+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c0af940 con 0x7f1d2c14be90 2024-03-25T19:29:03.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3406083147 0 0) 0x7f1d2c0af940 con 0x7f1d2c14be90 2024-03-25T19:29:03.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 >> v1:172.21.15.19:6790/0 conn(0x7f1d2c0b1730 legacy=0x7f1d2c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:03.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 >> v1:172.21.15.19:6789/0 conn(0x7f1d2c0b4d60 legacy=0x7f1d2c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:03.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d23fff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:03.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d23fff640 1 monclient: found mon.noname-c 2024-03-25T19:29:03.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:03.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d23fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:29:03.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c0b1030 con 0x7f1d2c14be90 2024-03-25T19:29:03.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d23fff640 10 monclient: _finish_auth 0 2024-03-25T19:29:03.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d23fff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:03.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:03.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d33582640 5 monclient: authenticate success, global_id 4174 2024-03-25T19:29:03.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.399+0000 7f1d33582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:03.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.400+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1d240014f0 con 0x7f1d2c14be90 2024-03-25T19:29:03.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.400+0000 7f1d23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:03.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.400+0000 7f1d23fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:03.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.400+0000 7f1d23fff640 10 monclient: dump: 2024-03-25T19:29:03.402 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:03.402 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:03.402 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:03.403 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:03.403 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:03.403 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:03.403 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:03.403 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:03.403 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:03.403 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:03.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.400+0000 7f1d33582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:03.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.402+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d240017c0 con 0x7f1d2c14be90 2024-03-25T19:29:03.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.402+0000 7f1d23fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:03.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.402+0000 7f1d23fff640 1 -- 172.21.15.19:0/4228259708 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1d24001b50 con 0x7f1d2c14be90 2024-03-25T19:29:03.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.402+0000 7f1d23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:03.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.402+0000 7f1d23fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:03.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.402+0000 7f1d23fff640 10 monclient: dump: 2024-03-25T19:29:03.407 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:03.407 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:03.407 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:03.407 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:03.407 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:03.407 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:03.407 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:03.407 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:03.407 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.402+0000 7f1d33582640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.402+0000 7f1d33582640 4 set_mon_vals no callback set 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.403+0000 7f1d33582640 10 monclient: shutdown 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.403+0000 7f1d33582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.403+0000 7f1d33582640 1 -- 172.21.15.19:0/4228259708 >> v1:172.21.15.188:6789/0 conn(0x7f1d2c14be90 legacy=0x7f1d2c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.404+0000 7f1d33582640 1 -- 172.21.15.19:0/4228259708 shutdown_connections 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.404+0000 7f1d33582640 1 -- 172.21.15.19:0/4228259708 >> 172.21.15.19:0/4228259708 conn(0x7f1d2c0617a0 msgr2=0x7f1d2c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.404+0000 7f1d33582640 1 -- 172.21.15.19:0/4228259708 shutdown_connections 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.404+0000 7f1d33582640 1 -- 172.21.15.19:0/4228259708 wait complete. 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.405+0000 7f1d33582640 10 monclient: build_initial_monmap 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.405+0000 7f1d33582640 10 monclient: monmap: 2024-03-25T19:29:03.408 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.406+0000 7f1d33582640 1 librados: starting msgr at 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.406+0000 7f1d33582640 1 librados: starting objecter 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.406+0000 7f1d33582640 1 Processor -- start 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.406+0000 7f1d33582640 1 -- start start 2024-03-25T19:29:03.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.406+0000 7f1d33582640 1 librados: setting wanted keys 2024-03-25T19:29:03.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.406+0000 7f1d33582640 1 librados: calling monclient init 2024-03-25T19:29:03.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.406+0000 7f1d33582640 10 monclient: init 2024-03-25T19:29:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.407+0000 7f1d33582640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.407+0000 7f1d33582640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.407+0000 7f1d33582640 10 monclient(hunting): picked mon.noname-c con 0x7f1d2c0b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.407+0000 7f1d33582640 10 monclient(hunting): picked mon.noname-b con 0x7f1d2c0b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.407+0000 7f1d33582640 10 monclient(hunting): picked mon.noname-a con 0x7f1d2c14be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.407+0000 7f1d33582640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c13b150 con 0x7f1d2c14be90 2024-03-25T19:29:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.408+0000 7f1d33582640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c13b580 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.408+0000 7f1d33582640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c13c240 con 0x7f1d2c0b1730 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.408+0000 7f1d33582640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.408+0000 7f1d33582640 10 monclient(hunting): authenticate will time out at 5996.678208s 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.408+0000 7f1d30b1d640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f1d2c0b4d60 0x7f1d2c1dc180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:37748/0 (socket says 172.21.15.19:37748) 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.408+0000 7f1d30b1d640 1 -- 172.21.15.19:0/1928294197 learned_addr learned my addr 172.21.15.19:0/1928294197 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.408+0000 7f1d21ffb640 10 client.?.objecter ms_handle_connect 0x7f1d2c0b4d60 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.408+0000 7f1d21ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.409+0000 7f1d21ffb640 10 client.?.objecter ms_handle_connect 0x7f1d2c0b1730 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.409+0000 7f1d21ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.409+0000 7f1d21ffb640 10 client.?.objecter ms_handle_connect 0x7f1d2c14be90 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.409+0000 7f1d21ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.412+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013438934 0 0) 0x7f1d2c13b580 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.412+0000 7f1d21ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:03.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.412+0000 7f1d21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:03.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.412+0000 7f1d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:03.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.412+0000 7f1d21ffb640 10 monclient(hunting): my global_id is 4177 2024-03-25T19:29:03.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.412+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d10004eb0 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.412+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072657193 0 0) 0x7f1d2c13c240 con 0x7f1d2c0b1730 2024-03-25T19:29:03.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.413+0000 7f1d21ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:03.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.413+0000 7f1d21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:03.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.413+0000 7f1d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:03.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.413+0000 7f1d21ffb640 10 monclient(hunting): my global_id is 4166 2024-03-25T19:29:03.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.413+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c13b580 con 0x7f1d2c0b1730 2024-03-25T19:29:03.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.413+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518495549 0 0) 0x7f1d2c13b150 con 0x7f1d2c14be90 2024-03-25T19:29:03.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.413+0000 7f1d21ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:03.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.413+0000 7f1d21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:03.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.413+0000 7f1d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:03.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.413+0000 7f1d21ffb640 10 monclient(hunting): my global_id is 4179 2024-03-25T19:29:03.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.413+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c13c240 con 0x7f1d2c14be90 2024-03-25T19:29:03.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.419+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2149474674 0 0) 0x7f1d10004eb0 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.419+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d2c13b150 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.419+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1d1c0019f0 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.419+0000 7f1d21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.419+0000 7f1d21ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.419+0000 7f1d21ffb640 10 monclient(hunting): dump: 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:03.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.419+0000 7f1d21ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:03.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.422+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3346975381 0 0) 0x7f1d2c13b580 con 0x7f1d2c0b1730 2024-03-25T19:29:03.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.422+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d10004eb0 con 0x7f1d2c0b1730 2024-03-25T19:29:03.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.422+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1d28001cb0 con 0x7f1d2c0b1730 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.422+0000 7f1d21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.422+0000 7f1d21ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.422+0000 7f1d21ffb640 10 monclient(hunting): dump: 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:03.423 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:03.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.423+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1907436603 0 0) 0x7f1d2c13c240 con 0x7f1d2c14be90 2024-03-25T19:29:03.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.423+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d2c13b580 con 0x7f1d2c14be90 2024-03-25T19:29:03.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.424+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1d24001b10 con 0x7f1d2c14be90 2024-03-25T19:29:03.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.424+0000 7f1d21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:03.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:03.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 10 monclient(hunting): dump: 2024-03-25T19:29:03.427 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:03.427 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:03.427 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:03.427 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:03.427 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:03.428 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:03.428 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:03.428 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:03.428 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:03.428 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:03.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2876633905 0 0) 0x7f1d2c13b150 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 >> v1:172.21.15.19:6790/0 conn(0x7f1d2c0b1730 legacy=0x7f1d2c13bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:03.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 >> v1:172.21.15.19:6789/0 conn(0x7f1d2c14be90 legacy=0x7f1d2c0716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:03.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:29:03.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 1 monclient: found mon.b 2024-03-25T19:29:03.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:03.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c1dc890 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 10 monclient: _finish_auth 0 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.425+0000 7f1d33582640 5 monclient: authenticate success, global_id 4177 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.426+0000 7f1d33582640 10 monclient: _renew_subs 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.426+0000 7f1d33582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.426+0000 7f1d33582640 1 -- 172.21.15.19:0/1928294197 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2c13c520 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.426+0000 7f1d33582640 10 client.4177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.426+0000 7f1d33582640 10 monclient: _renew_subs 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.426+0000 7f1d33582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.426+0000 7f1d33582640 1 -- 172.21.15.19:0/1928294197 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2c071f90 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.426+0000 7f1d33582640 1 librados: init done 2024-03-25T19:29:03.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.428+0000 7f1d33582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:03.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.428+0000 7f1d33582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:03.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.428+0000 7f1d33582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:03.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.428+0000 7f1d33582640 1 -- 172.21.15.19:0/1928294197 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2c0b22c0 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.428+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d1c0047e0 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.428+0000 7f1d21ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:03.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.428+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1d1c004af0 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.428+0000 7f1d21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:03.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.428+0000 7f1d21ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:03.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.428+0000 7f1d21ffb640 10 monclient: dump: 2024-03-25T19:29:03.432 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:03.432 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:03.432 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:03.432 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:03.432 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:03.432 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:03.432 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:03.432 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:03.432 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:03.432 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:03.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.428+0000 7f1d237fe640 4 set_mon_vals no callback set 2024-03-25T19:29:03.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.453+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f1d1c01aac0 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.454+0000 7f1d3131e640 10 monclient: get_auth_request con 0x7f1d1006e060 auth_method 0 2024-03-25T19:29:03.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.454+0000 7f1d21ffb640 10 client.4177.objecter ms_handle_connect 0x7f1d1006e060 2024-03-25T19:29:03.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.457+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f1d1c0547e0 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.457+0000 7f1d21ffb640 10 client.4177.objecter ms_dispatch 0x7f1d2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:03.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.457+0000 7f1d21ffb640 3 client.4177.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:03.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.457+0000 7f1d21ffb640 3 client.4177.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:03.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.457+0000 7f1d21ffb640 20 client.4177.objecter dump_active .. 0 homeless 2024-03-25T19:29:03.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.881+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d1c050120 con 0x7f1d2c0b4d60 2024-03-25T19:29:03.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.881+0000 7f1d21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:03.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:03.881+0000 7f1d21ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:04.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.262+0000 7f1d33582640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:04.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.262+0000 7f1d33582640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:04.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.262+0000 7f1d33582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:04.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.262+0000 7f1d33582640 1 -- 172.21.15.19:0/1928294197 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d2c13c6b0 con 0x7f1d2c0b4d60 2024-03-25T19:29:04.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.335+0000 7f1d21ffb640 1 -- 172.21.15.19:0/1928294197 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f1d1c052220 con 0x7f1d2c0b4d60 2024-03-25T19:29:04.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.335+0000 7f1d21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:04.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.335+0000 7f1d21ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:04.336 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:29:04.336 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:04.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.338+0000 7f1d33582640 10 librados: watch_flush enter 2024-03-25T19:29:04.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.338+0000 7f1d33582640 10 librados: watch_flush exit 2024-03-25T19:29:04.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.338+0000 7f1d33582640 20 client.4177.objecter shutdown clearing up homeless session... 2024-03-25T19:29:04.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.338+0000 7f1d33582640 10 client.4177.objecter successfully canceled tick 2024-03-25T19:29:04.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.338+0000 7f1d33582640 1 -- 172.21.15.19:0/1928294197 >> v1:172.21.15.188:6800/2207506229 conn(0x7f1d1006e060 legacy=0x7f1d10070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:04.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.339+0000 7f1d33582640 10 monclient: shutdown 2024-03-25T19:29:04.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.339+0000 7f1d33582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:04.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.339+0000 7f1d33582640 1 -- 172.21.15.19:0/1928294197 >> v1:172.21.15.188:6789/0 conn(0x7f1d2c0b4d60 legacy=0x7f1d2c1dc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:04.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.339+0000 7f1d33582640 1 -- 172.21.15.19:0/1928294197 shutdown_connections 2024-03-25T19:29:04.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.339+0000 7f1d33582640 1 -- 172.21.15.19:0/1928294197 >> 172.21.15.19:0/1928294197 conn(0x7f1d2c0617a0 msgr2=0x7f1d2c0b62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:04.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.339+0000 7f1d33582640 1 -- 172.21.15.19:0/1928294197 shutdown_connections 2024-03-25T19:29:04.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.339+0000 7f1d33582640 1 -- 172.21.15.19:0/1928294197 wait complete. 2024-03-25T19:29:04.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:04.339+0000 7f1d33582640 1 librados: shutdown 2024-03-25T19:29:04.352 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:07.724 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:06.679 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:07.792 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:06.748 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:07.853 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:06.808 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:07.914 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:06.869 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.072 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:07.014 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.132 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:07.074 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.142 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:07.106 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.201 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:07.165 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.229 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:07.214 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.260 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:07.234 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.290 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:07.276 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.324 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:07.297 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.337 INFO:tasks.ceph.osd.5.smithi188.stderr:2024-03-25T19:29:08.331+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:29:08.340 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:07.316 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.401 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:07.377 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.463 INFO:tasks.ceph.osd.6.smithi188.stderr:2024-03-25T19:29:08.457+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:29:08.560 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:07.549 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.624 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:07.612 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.679 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-03-25T19:29:08.674+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:29:08.801 INFO:tasks.ceph.osd.4.smithi188.stderr:2024-03-25T19:29:08.792+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:29:08.824 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:07.779 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.834 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-03-25T19:29:08.828+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:29:08.879 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-03-25T19:29:08.874+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:29:08.940 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:07.895 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:08.973 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-03-25T19:29:08.964+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:29:09.167 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:08.108 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.196 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:08.151 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.218 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-03-25T19:29:09.212+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:29:09.258 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:08.222 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.305 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:08.291 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.315 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:08.271 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.359 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:08.332 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.471 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:08.448 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.529 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:08.484 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.531 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:08.473 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.629 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:08.593 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.669 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:08.624 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.670 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:08.656 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.701 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:08.689 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.731 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:08.705 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.793 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:08.748 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.836 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:08.813 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.890 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:08.832 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.946 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:08.910 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.970 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:08.925 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:09.994 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:08.980 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.057 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:09.012 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.079 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:09.068 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.094 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:09.068 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.178 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:09.120 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.196 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:09.172 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.212 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:09.176 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.261 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:09.216 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.262 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:09.248 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.325 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:09.280 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.388 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:09.361 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.405 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:09.394 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.465 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:09.406 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.481 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:09.444 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.485 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:09.462 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.537 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:09.523 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.548 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:09.503 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.594 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:09.549 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.671 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:09.659 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.684 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:09.657 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.751 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:09.715 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.756 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:09.732 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.757 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:09.699 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.808 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:09.794 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.839 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:09.795 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.870 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:09.825 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.935 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:09.923 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:10.973 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:09.946 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.023 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:09.987 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.049 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:10.025 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.055 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:09.996 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.076 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:10.062 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.128 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:10.083 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.139 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:10.094 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.158 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:11.207 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:10.196 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.247+0000 7fb1ebc9c640 10 monclient: get_monmap_and_config 2024-03-25T19:29:11.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.247+0000 7fb1ebc9c640 10 monclient: build_initial_monmap 2024-03-25T19:29:11.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.247+0000 7fb1ebc9c640 10 monclient: monmap: 2024-03-25T19:29:11.247 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:11.247 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:11.247 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:29:11.248798+0000 2024-03-25T19:29:11.248 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:29:11.248798+0000 2024-03-25T19:29:11.248 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:11.248 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:11.248 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:11.248 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:29:11.248 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:29:11.248 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:11.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 1 Processor -- start 2024-03-25T19:29:11.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 1 -- start start 2024-03-25T19:29:11.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 10 monclient: init 2024-03-25T19:29:11.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:11.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:11.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e406c380 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:11.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e406d670 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:11.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e40af8f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:11.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e406c9a0 con 0x7fb1e406d670 2024-03-25T19:29:11.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e40ada90 con 0x7fb1e406c380 2024-03-25T19:29:11.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e40b0d10 con 0x7fb1e40af8f0 2024-03-25T19:29:11.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:11.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.248+0000 7fb1ebc9c640 10 monclient(hunting): authenticate will time out at 6004.518821s 2024-03-25T19:29:11.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.249+0000 7fb1ea239640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fb1e40af8f0 0x7fb1e40aedf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:52674/0 (socket says 172.21.15.19:52674) 2024-03-25T19:29:11.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.249+0000 7fb1e9237640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fb1e406d670 0x7fb1e40ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:40394/0 (socket says 172.21.15.19:40394) 2024-03-25T19:29:11.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.249+0000 7fb1ea239640 1 -- 172.21.15.19:0/1924769851 learned_addr learned my addr 172.21.15.19:0/1924769851 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:11.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.251+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609666918 0 0) 0x7fb1e40ada90 con 0x7fb1e406c380 2024-03-25T19:29:11.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.251+0000 7fb1e8a36640 10 monclient(hunting): _negotiate 2024-03-25T19:29:11.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.251+0000 7fb1e8a36640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:11.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.251+0000 7fb1e8a36640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:11.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.251+0000 7fb1e8a36640 10 monclient(hunting): my global_id is 4204 2024-03-25T19:29:11.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.251+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d0003520 con 0x7fb1e406c380 2024-03-25T19:29:11.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.252+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344788667 0 0) 0x7fb1e406c9a0 con 0x7fb1e406d670 2024-03-25T19:29:11.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.252+0000 7fb1e8a36640 10 monclient(hunting): _negotiate 2024-03-25T19:29:11.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.252+0000 7fb1e8a36640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:11.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.252+0000 7fb1e8a36640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:11.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.252+0000 7fb1e8a36640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:29:11.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.252+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e40ada90 con 0x7fb1e406d670 2024-03-25T19:29:11.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.252+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756386256 0 0) 0x7fb1e40b0d10 con 0x7fb1e40af8f0 2024-03-25T19:29:11.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.252+0000 7fb1e8a36640 10 monclient(hunting): _negotiate 2024-03-25T19:29:11.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.252+0000 7fb1e8a36640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:11.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.252+0000 7fb1e8a36640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:11.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.252+0000 7fb1e8a36640 10 monclient(hunting): my global_id is 4193 2024-03-25T19:29:11.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.252+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e406c9a0 con 0x7fb1e40af8f0 2024-03-25T19:29:11.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3297602741 0 0) 0x7fb1d0003520 con 0x7fb1e406c380 2024-03-25T19:29:11.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 >> v1:172.21.15.19:6790/0 conn(0x7fb1e40af8f0 legacy=0x7fb1e40aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:11.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 >> v1:172.21.15.19:6789/0 conn(0x7fb1e406d670 legacy=0x7fb1e40ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:11.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 10 monclient: _finish_hunting 0 2024-03-25T19:29:11.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 1 monclient: found mon.noname-c 2024-03-25T19:29:11.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:11.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:29:11.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e40add00 con 0x7fb1e406c380 2024-03-25T19:29:11.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 10 monclient: _finish_auth 0 2024-03-25T19:29:11.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 10 monclient: _check_auth_tickets 2024-03-25T19:29:11.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:11.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb1e0001690 con 0x7fb1e406c380 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.254+0000 7fb1e8a36640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.255+0000 7fb1e8a36640 10 monclient: dump: 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.255+0000 7fb1ebc9c640 5 monclient: authenticate success, global_id 4204 2024-03-25T19:29:11.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.255+0000 7fb1ebc9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:11.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.256+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1e00017f0 con 0x7fb1e406c380 2024-03-25T19:29:11.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.256+0000 7fb1e8a36640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:11.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.256+0000 7fb1e8a36640 1 -- 172.21.15.19:0/1924769851 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb1e0001b80 con 0x7fb1e406c380 2024-03-25T19:29:11.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.256+0000 7fb1ebc9c640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:11.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.256+0000 7fb1ebc9c640 4 set_mon_vals no callback set 2024-03-25T19:29:11.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.256+0000 7fb1e8a36640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:11.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.256+0000 7fb1e8a36640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.256+0000 7fb1e8a36640 10 monclient: dump: 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.257+0000 7fb1ebc9c640 10 monclient: shutdown 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.257+0000 7fb1ebc9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.257+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/1924769851 >> v1:172.21.15.188:6789/0 conn(0x7fb1e406c380 legacy=0x7fb1e40b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:11.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.257+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/1924769851 shutdown_connections 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.257+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/1924769851 >> 172.21.15.19:0/1924769851 conn(0x7fb1e40b5c00 msgr2=0x7fb1e40b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.257+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/1924769851 shutdown_connections 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.257+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/1924769851 wait complete. 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.258+0000 7fb1ebc9c640 10 monclient: build_initial_monmap 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.258+0000 7fb1ebc9c640 10 monclient: monmap: 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:29:11.260 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:11.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.258+0000 7fb1ebc9c640 1 librados: starting msgr at 2024-03-25T19:29:11.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.258+0000 7fb1ebc9c640 1 librados: starting objecter 2024-03-25T19:29:11.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.258+0000 7fb1ebc9c640 1 Processor -- start 2024-03-25T19:29:11.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.258+0000 7fb1ebc9c640 1 -- start start 2024-03-25T19:29:11.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.258+0000 7fb1ebc9c640 1 librados: setting wanted keys 2024-03-25T19:29:11.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.258+0000 7fb1ebc9c640 1 librados: calling monclient init 2024-03-25T19:29:11.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.258+0000 7fb1ebc9c640 10 monclient: init 2024-03-25T19:29:11.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.259+0000 7fb1ebc9c640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:11.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.259+0000 7fb1ebc9c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:11.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.259+0000 7fb1ebc9c640 10 monclient(hunting): picked mon.noname-b con 0x7fb1e406c380 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:11.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.259+0000 7fb1ebc9c640 10 monclient(hunting): picked mon.noname-c con 0x7fb1e406d670 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.259+0000 7fb1ebc9c640 10 monclient(hunting): picked mon.noname-a con 0x7fb1e40af8f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.259+0000 7fb1ebc9c640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e4143390 con 0x7fb1e40af8f0 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.259+0000 7fb1ebc9c640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e4148410 con 0x7fb1e406c380 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.259+0000 7fb1ebc9c640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e413d6d0 con 0x7fb1e406d670 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.259+0000 7fb1ebc9c640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.259+0000 7fb1ebc9c640 10 monclient(hunting): authenticate will time out at 6004.529160s 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.259+0000 7fb1ea239640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fb1e40af8f0 0x7fb1e413cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:40398/0 (socket says 172.21.15.19:40398) 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.259+0000 7fb1ea239640 1 -- 172.21.15.19:0/2323708885 learned_addr learned my addr 172.21.15.19:0/2323708885 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.260+0000 7fb1da7fc640 10 client.?.objecter ms_handle_connect 0x7fb1e40af8f0 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.260+0000 7fb1da7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.260+0000 7fb1da7fc640 10 client.?.objecter ms_handle_connect 0x7fb1e406d670 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.260+0000 7fb1da7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.260+0000 7fb1da7fc640 10 client.?.objecter ms_handle_connect 0x7fb1e406c380 2024-03-25T19:29:11.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.260+0000 7fb1da7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.261+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578228988 0 0) 0x7fb1e4143390 con 0x7fb1e40af8f0 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.261+0000 7fb1da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.261+0000 7fb1da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.261+0000 7fb1da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.261+0000 7fb1da7fc640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c8004eb0 con 0x7fb1e40af8f0 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861895252 0 0) 0x7fb1e4148410 con 0x7fb1e406c380 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e4143390 con 0x7fb1e406c380 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593944730 0 0) 0x7fb1e413d6d0 con 0x7fb1e406d670 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:11.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:11.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:11.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 10 monclient(hunting): my global_id is 4196 2024-03-25T19:29:11.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.262+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e4148410 con 0x7fb1e406d670 2024-03-25T19:29:11.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2778411499 0 0) 0x7fb1e4143390 con 0x7fb1e406c380 2024-03-25T19:29:11.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1e413d6d0 con 0x7fb1e406c380 2024-03-25T19:29:11.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb1e0001960 con 0x7fb1e406c380 2024-03-25T19:29:11.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:11.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:11.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 10 monclient(hunting): dump: 2024-03-25T19:29:11.269 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:11.269 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:11.269 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:11.269 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4291615424 0 0) 0x7fb1c8004eb0 con 0x7fb1e40af8f0 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1e4143390 con 0x7fb1e40af8f0 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb1dc001b00 con 0x7fb1e40af8f0 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:11.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.266+0000 7fb1da7fc640 10 monclient(hunting): dump: 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.267+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2959410424 0 0) 0x7fb1e4148410 con 0x7fb1e406d670 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.267+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1c8004eb0 con 0x7fb1e406d670 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.267+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb1d40018f0 con 0x7fb1e406d670 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.267+0000 7fb1da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.267+0000 7fb1da7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:11.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.267+0000 7fb1da7fc640 10 monclient(hunting): dump: 2024-03-25T19:29:11.272 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:11.272 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:11.272 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:11.272 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:11.272 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:11.272 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:11.272 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:11.272 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:11.272 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:11.272 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:11.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2156635496 0 0) 0x7fb1e413d6d0 con 0x7fb1e406c380 2024-03-25T19:29:11.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 >> v1:172.21.15.19:6790/0 conn(0x7fb1e406d670 legacy=0x7fb1e41da7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:11.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 >> v1:172.21.15.19:6789/0 conn(0x7fb1e40af8f0 legacy=0x7fb1e413cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:11.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1da7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1da7fc640 1 monclient: found mon.b 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1da7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e4142f60 con 0x7fb1e406c380 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1da7fc640 10 monclient: _finish_auth 0 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1da7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1ebc9c640 5 monclient: authenticate success, global_id 4207 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1ebc9c640 10 monclient: _renew_subs 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1ebc9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.269+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/2323708885 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e4147530 con 0x7fb1e406c380 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.270+0000 7fb1ebc9c640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.270+0000 7fb1ebc9c640 10 monclient: _renew_subs 2024-03-25T19:29:11.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.270+0000 7fb1ebc9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:11.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.270+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/2323708885 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e41438b0 con 0x7fb1e406c380 2024-03-25T19:29:11.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.270+0000 7fb1ebc9c640 1 librados: init done 2024-03-25T19:29:11.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.270+0000 7fb1ebc9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:11.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.270+0000 7fb1ebc9c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:11.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.270+0000 7fb1ebc9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:11.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.270+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/2323708885 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e4143540 con 0x7fb1e406c380 2024-03-25T19:29:11.282 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:10.255 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.283+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1e0001b80 con 0x7fb1e406c380 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.283+0000 7fb1da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.283+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb1e00038a0 con 0x7fb1e406c380 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.283+0000 7fb1da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.283+0000 7fb1da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.283+0000 7fb1da7fc640 10 monclient: dump: 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:11.284 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:11.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.283+0000 7fb1dbfff640 4 set_mon_vals no callback set 2024-03-25T19:29:11.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.284+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fb1e0019870 con 0x7fb1e406c380 2024-03-25T19:29:11.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.285+0000 7fb1e9237640 10 monclient: get_auth_request con 0x7fb1c806e0f0 auth_method 0 2024-03-25T19:29:11.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.285+0000 7fb1da7fc640 10 client.4207.objecter ms_handle_connect 0x7fb1c806e0f0 2024-03-25T19:29:11.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.287+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fb1e00533f0 con 0x7fb1e406c380 2024-03-25T19:29:11.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.287+0000 7fb1da7fc640 10 client.4207.objecter ms_dispatch 0x7fb1e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:11.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.287+0000 7fb1da7fc640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:11.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.287+0000 7fb1da7fc640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:11.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.287+0000 7fb1da7fc640 20 client.4207.objecter dump_active .. 0 homeless 2024-03-25T19:29:11.306 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:10.269 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.342 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:10.283 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.344 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:10.330 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.349 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:10.325 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.431 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:10.387 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.471 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:10.459 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.517+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1e004ef00 con 0x7fb1e406c380 2024-03-25T19:29:11.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.517+0000 7fb1da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:11.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.517+0000 7fb1da7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:11.576 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:10.549 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.584 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:10.548 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.607 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:10.562 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.611 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:10.597 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.617 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:10.594 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.630 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:10.572 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.679 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:10.634 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.698+0000 7fb1ebc9c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:11.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.698+0000 7fb1ebc9c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:11.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.698+0000 7fb1ebc9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:11.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.698+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/2323708885 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1e40627b0 con 0x7fb1e406c380 2024-03-25T19:29:11.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.712+0000 7fb1da7fc640 1 -- 172.21.15.19:0/2323708885 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fb1e0050e30 con 0x7fb1e406c380 2024-03-25T19:29:11.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.712+0000 7fb1da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:11.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.712+0000 7fb1da7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:11.712 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:29:11.712 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:11.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 10 librados: watch_flush enter 2024-03-25T19:29:11.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 10 librados: watch_flush exit 2024-03-25T19:29:11.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 20 client.4207.objecter shutdown clearing up homeless session... 2024-03-25T19:29:11.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 10 client.4207.objecter successfully canceled tick 2024-03-25T19:29:11.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/2323708885 >> v1:172.21.15.188:6800/2207506229 conn(0x7fb1c806e0f0 legacy=0x7fb1c80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:11.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 10 monclient: shutdown 2024-03-25T19:29:11.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:11.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/2323708885 >> v1:172.21.15.188:6789/0 conn(0x7fb1e406c380 legacy=0x7fb1e41452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:11.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/2323708885 shutdown_connections 2024-03-25T19:29:11.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/2323708885 >> 172.21.15.19:0/2323708885 conn(0x7fb1e40b5c00 msgr2=0x7fb1e4070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:11.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/2323708885 shutdown_connections 2024-03-25T19:29:11.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 1 -- 172.21.15.19:0/2323708885 wait complete. 2024-03-25T19:29:11.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:11.714+0000 7fb1ebc9c640 1 librados: shutdown 2024-03-25T19:29:11.723 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:11.754 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:10.743 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.875 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:10.848 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.896 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:10.872 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:11.959 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:10.914 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.013 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:10.969 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.022 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:11.010 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.077 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:11.038 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.094 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:11.080 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.129 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:11.093 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.149 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:11.136 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.222 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:11.164 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.280 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:11.221 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.404 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:11.377 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.423 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:11.399 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.473 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:11.446 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.478 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:11.455 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.563 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:11.551 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:12.619 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:11.607 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:17.579 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:16.534 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:17.594 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:16.580 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:17.837 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:16.800 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:17.918 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:16.873 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:17.982 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:16.958 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:18.131 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:17.073 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:18.187 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:17.160 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:18.343 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:17.331 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:18.529 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:18.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.654+0000 7facd27c8640 10 monclient: get_monmap_and_config 2024-03-25T19:29:18.655 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.654+0000 7facd27c8640 10 monclient: build_initial_monmap 2024-03-25T19:29:18.655 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.654+0000 7facd27c8640 10 monclient: monmap: 2024-03-25T19:29:18.655 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:18.655 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:18.655 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:29:18.656095+0000 2024-03-25T19:29:18.655 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:29:18.656095+0000 2024-03-25T19:29:18.655 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:18.655 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:18.655 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:18.655 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:29:18.655 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:29:18.655 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:18.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.655+0000 7facd27c8640 1 Processor -- start 2024-03-25T19:29:18.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.655+0000 7facd27c8640 1 -- start start 2024-03-25T19:29:18.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.655+0000 7facd27c8640 10 monclient: init 2024-03-25T19:29:18.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.656+0000 7facd27c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:18.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.656+0000 7facd27c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:18.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.656+0000 7facd27c8640 10 monclient(hunting): picked mon.noname-b con 0x7faccc0b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:18.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.656+0000 7facd27c8640 10 monclient(hunting): picked mon.noname-c con 0x7faccc0b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:18.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.656+0000 7facd27c8640 10 monclient(hunting): picked mon.noname-a con 0x7faccc14be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:18.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.656+0000 7facd27c8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0af940 con 0x7faccc14be90 2024-03-25T19:29:18.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.656+0000 7facd27c8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0b0a70 con 0x7faccc0b1730 2024-03-25T19:29:18.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.657+0000 7facd27c8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc0b0d50 con 0x7faccc0b4d60 2024-03-25T19:29:18.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.657+0000 7facd27c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:18.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.657+0000 7facd27c8640 10 monclient(hunting): authenticate will time out at 6011.927328s 2024-03-25T19:29:18.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.657+0000 7faccb7fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7faccc0b1730 0x7faccc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:50884/0 (socket says 172.21.15.19:50884) 2024-03-25T19:29:18.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.657+0000 7faccb7fe640 1 -- 172.21.15.19:0/182058870 learned_addr learned my addr 172.21.15.19:0/182058870 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:18.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898001291 0 0) 0x7faccc0b0a70 con 0x7faccc0b1730 2024-03-25T19:29:18.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:18.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:18.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:18.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:29:18.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb4003520 con 0x7faccc0b1730 2024-03-25T19:29:18.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332677881 0 0) 0x7faccc0b0d50 con 0x7faccc0b4d60 2024-03-25T19:29:18.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:18.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:18.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:18.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 10 monclient(hunting): my global_id is 4199 2024-03-25T19:29:18.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faccc0b0a70 con 0x7faccc0b4d60 2024-03-25T19:29:18.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240080966 0 0) 0x7faccc0af940 con 0x7faccc14be90 2024-03-25T19:29:18.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:18.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:18.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:18.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:29:18.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.661+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faccc0b0d50 con 0x7faccc14be90 2024-03-25T19:29:18.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3849059312 0 0) 0x7faccc0b0a70 con 0x7faccc0b4d60 2024-03-25T19:29:18.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 >> v1:172.21.15.188:6789/0 conn(0x7faccc0b1730 legacy=0x7faccc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 >> v1:172.21.15.19:6789/0 conn(0x7faccc14be90 legacy=0x7faccc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:18.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 1 monclient: found mon.noname-b 2024-03-25T19:29:18.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc0b1030 con 0x7faccc0b4d60 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7facd27c8640 5 monclient: authenticate success, global_id 4199 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7facd27c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7facbc001630 con 0x7faccc0b4d60 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7faccaffd640 10 monclient: dump: 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:18.667 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:18.668 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:18.668 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:18.668 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:18.668 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:18.668 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:18.668 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:18.668 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:18.668 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:18.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.666+0000 7facd27c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:18.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.672+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facbc001880 con 0x7faccc0b4d60 2024-03-25T19:29:18.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.672+0000 7faccaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:18.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.672+0000 7facd27c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:18.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.672+0000 7facd27c8640 4 set_mon_vals no callback set 2024-03-25T19:29:18.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.672+0000 7facd27c8640 10 monclient: shutdown 2024-03-25T19:29:18.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.672+0000 7facd27c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:18.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.672+0000 7facd27c8640 1 -- 172.21.15.19:0/182058870 >> v1:172.21.15.19:6790/0 conn(0x7faccc0b4d60 legacy=0x7faccc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.672+0000 7faccaffd640 1 -- 172.21.15.19:0/182058870 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7facbc001630 con 0x7faccc0b4d60 2024-03-25T19:29:18.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.672+0000 7faccaffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-25T19:29:18.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.673+0000 7facd27c8640 1 -- 172.21.15.19:0/182058870 shutdown_connections 2024-03-25T19:29:18.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.673+0000 7facd27c8640 1 -- 172.21.15.19:0/182058870 >> 172.21.15.19:0/182058870 conn(0x7faccc0617a0 msgr2=0x7faccc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:18.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.673+0000 7facd27c8640 1 -- 172.21.15.19:0/182058870 shutdown_connections 2024-03-25T19:29:18.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.673+0000 7facd27c8640 1 -- 172.21.15.19:0/182058870 wait complete. 2024-03-25T19:29:18.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.674+0000 7facd27c8640 10 monclient: build_initial_monmap 2024-03-25T19:29:18.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.674+0000 7facd27c8640 10 monclient: monmap: 2024-03-25T19:29:18.674 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:18.674 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:18.674 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:29:18.674 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:29:18.674 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:18.674 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:29:18.674 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:18.674 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:29:18.674 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:29:18.674 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.674+0000 7facd27c8640 1 librados: starting msgr at 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.674+0000 7facd27c8640 1 librados: starting objecter 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.674+0000 7facd27c8640 1 Processor -- start 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.675+0000 7facd27c8640 1 -- start start 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.675+0000 7facd27c8640 1 librados: setting wanted keys 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.675+0000 7facd27c8640 1 librados: calling monclient init 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.675+0000 7facd27c8640 10 monclient: init 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.676+0000 7facd27c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.676+0000 7facd27c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.676+0000 7facd27c8640 10 monclient(hunting): picked mon.noname-c con 0x7faccc0b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.676+0000 7facd27c8640 10 monclient(hunting): picked mon.noname-b con 0x7faccc0b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.676+0000 7facd27c8640 10 monclient(hunting): picked mon.noname-a con 0x7faccc14be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:18.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.676+0000 7facd27c8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc1dabd0 con 0x7faccc14be90 2024-03-25T19:29:18.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.676+0000 7facd27c8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc143900 con 0x7faccc0b4d60 2024-03-25T19:29:18.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.676+0000 7facd27c8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faccc135d40 con 0x7faccc0b1730 2024-03-25T19:29:18.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.676+0000 7facd27c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:18.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.677+0000 7facd27c8640 10 monclient(hunting): authenticate will time out at 6011.946977s 2024-03-25T19:29:18.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.677+0000 7faccbfff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7faccc0b1730 0x7faccc1367a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:52706/0 (socket says 172.21.15.19:52706) 2024-03-25T19:29:18.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.677+0000 7faccb7fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7faccc0b4d60 0x7faccc1db060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:50898/0 (socket says 172.21.15.19:50898) 2024-03-25T19:29:18.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.677+0000 7faccb7fe640 1 -- 172.21.15.19:0/1317702472 learned_addr learned my addr 172.21.15.19:0/1317702472 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:18.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.677+0000 7facc8ff9640 10 client.?.objecter ms_handle_connect 0x7faccc0b4d60 2024-03-25T19:29:18.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.677+0000 7facc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:18.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.677+0000 7facc8ff9640 10 client.?.objecter ms_handle_connect 0x7faccc0b1730 2024-03-25T19:29:18.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.677+0000 7facc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:18.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.680+0000 7facc8ff9640 10 client.?.objecter ms_handle_connect 0x7faccc14be90 2024-03-25T19:29:18.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.680+0000 7facc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:18.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.680+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988075107 0 0) 0x7faccc135d40 con 0x7faccc0b1730 2024-03-25T19:29:18.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.680+0000 7facc8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:18.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.680+0000 7facc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:18.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.680+0000 7facc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:18.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.680+0000 7facc8ff9640 10 monclient(hunting): my global_id is 4202 2024-03-25T19:29:18.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.680+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb8004eb0 con 0x7faccc0b1730 2024-03-25T19:29:18.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.681+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276239235 0 0) 0x7faccc143900 con 0x7faccc0b4d60 2024-03-25T19:29:18.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.681+0000 7facc8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:18.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.681+0000 7facc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:18.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.681+0000 7facc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:18.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.681+0000 7facc8ff9640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:29:18.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.681+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb80051d0 con 0x7faccc0b4d60 2024-03-25T19:29:18.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.687+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2764117784 0 0) 0x7facb80051d0 con 0x7faccc0b4d60 2024-03-25T19:29:18.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.687+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faccc143900 con 0x7faccc0b4d60 2024-03-25T19:29:18.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.687+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7facc0001950 con 0x7faccc0b4d60 2024-03-25T19:29:18.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.687+0000 7facc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:18.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.687+0000 7facc8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:18.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.687+0000 7facc8ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:18.687 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:18.687 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:18.687 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:18.688 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:18.688 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:18.688 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:18.688 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:18.688 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:18.688 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:18.688 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:18.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.687+0000 7facc8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:18.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.687+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548143673 0 0) 0x7faccc1dabd0 con 0x7faccc14be90 2024-03-25T19:29:18.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.688+0000 7facc8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:18.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.688+0000 7facc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:18.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.688+0000 7facc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:18.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.688+0000 7facc8ff9640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:29:18.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.688+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb80051d0 con 0x7faccc14be90 2024-03-25T19:29:18.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.690+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 824370013 0 0) 0x7facb8004eb0 con 0x7faccc0b1730 2024-03-25T19:29:18.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.690+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faccc1dabd0 con 0x7faccc0b1730 2024-03-25T19:29:18.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.690+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7facbc001d30 con 0x7faccc0b1730 2024-03-25T19:29:18.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.690+0000 7facc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:18.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.690+0000 7facc8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:18.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.690+0000 7facc8ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:18.690 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:18.691 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:18.691 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:18.691 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:18.691 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:18.691 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:18.691 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:18.691 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:18.691 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:18.691 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:18.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2782978145 0 0) 0x7faccc143900 con 0x7faccc0b4d60 2024-03-25T19:29:18.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 >> v1:172.21.15.19:6790/0 conn(0x7faccc0b1730 legacy=0x7faccc1367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 >> v1:172.21.15.19:6789/0 conn(0x7faccc14be90 legacy=0x7faccc12fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facc8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:29:18.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facc8ff9640 1 monclient: found mon.b 2024-03-25T19:29:18.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facc8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:18.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facc8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:18.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc136eb0 con 0x7faccc0b4d60 2024-03-25T19:29:18.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facc8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:29:18.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facc8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:29:18.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:18.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facd27c8640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:29:18.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facd27c8640 10 monclient: _renew_subs 2024-03-25T19:29:18.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facd27c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:18.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facd27c8640 1 -- 172.21.15.19:0/1317702472 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faccc138710 con 0x7faccc0b4d60 2024-03-25T19:29:18.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facd27c8640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:18.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facd27c8640 10 monclient: _renew_subs 2024-03-25T19:29:18.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facd27c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:18.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facd27c8640 1 -- 172.21.15.19:0/1317702472 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faccc1db950 con 0x7faccc0b4d60 2024-03-25T19:29:18.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.691+0000 7facd27c8640 1 librados: init done 2024-03-25T19:29:18.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.692+0000 7facd27c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:18.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.692+0000 7facd27c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:18.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.692+0000 7facd27c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:18.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.692+0000 7facd27c8640 1 -- 172.21.15.19:0/1317702472 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faccc0627b0 con 0x7faccc0b4d60 2024-03-25T19:29:18.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.698+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facc00047e0 con 0x7faccc0b4d60 2024-03-25T19:29:18.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.698+0000 7facc8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:18.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.698+0000 7facca7fc640 4 set_mon_vals no callback set 2024-03-25T19:29:18.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.714+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7facc0004af0 con 0x7faccc0b4d60 2024-03-25T19:29:18.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.714+0000 7facc8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:18.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.714+0000 7facc8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:18.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.714+0000 7facc8ff9640 10 monclient: dump: 2024-03-25T19:29:18.714 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:18.714 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:18.715 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:18.715 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:18.715 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:18.715 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:18.715 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:18.715 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:18.715 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:18.715 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:18.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.715+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7facc001aac0 con 0x7faccc0b4d60 2024-03-25T19:29:18.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.716+0000 7faccbfff640 10 monclient: get_auth_request con 0x7facb806e0d0 auth_method 0 2024-03-25T19:29:18.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.716+0000 7facc8ff9640 10 client.4213.objecter ms_handle_connect 0x7facb806e0d0 2024-03-25T19:29:18.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.718+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7facc0054850 con 0x7faccc0b4d60 2024-03-25T19:29:18.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.718+0000 7facc8ff9640 10 client.4213.objecter ms_dispatch 0x7faccc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:18.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.719+0000 7facc8ff9640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:18.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.719+0000 7facc8ff9640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:18.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:18.719+0000 7facc8ff9640 20 client.4213.objecter dump_active .. 0 homeless 2024-03-25T19:29:19.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.043+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facc0050190 con 0x7faccc0b4d60 2024-03-25T19:29:19.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.043+0000 7facc8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:19.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.043+0000 7facc8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:19.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.344+0000 7facd27c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:19.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.344+0000 7facd27c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:19.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.344+0000 7facd27c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:19.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.344+0000 7facd27c8640 1 -- 172.21.15.19:0/1317702472 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faccc069f40 con 0x7faccc0b4d60 2024-03-25T19:29:19.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.362+0000 7facc8ff9640 1 -- 172.21.15.19:0/1317702472 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7facc0052290 con 0x7faccc0b4d60 2024-03-25T19:29:19.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.362+0000 7facc8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:19.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.362+0000 7facc8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:19.363 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:29:19.363 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:19.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 10 librados: watch_flush enter 2024-03-25T19:29:19.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 10 librados: watch_flush exit 2024-03-25T19:29:19.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 20 client.4213.objecter shutdown clearing up homeless session... 2024-03-25T19:29:19.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 10 client.4213.objecter successfully canceled tick 2024-03-25T19:29:19.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 1 -- 172.21.15.19:0/1317702472 >> v1:172.21.15.188:6800/2207506229 conn(0x7facb806e0d0 legacy=0x7facb8070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:19.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 10 monclient: shutdown 2024-03-25T19:29:19.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:19.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 1 -- 172.21.15.19:0/1317702472 >> v1:172.21.15.188:6789/0 conn(0x7faccc0b4d60 legacy=0x7faccc1db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:19.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 1 -- 172.21.15.19:0/1317702472 shutdown_connections 2024-03-25T19:29:19.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 1 -- 172.21.15.19:0/1317702472 >> 172.21.15.19:0/1317702472 conn(0x7faccc0617a0 msgr2=0x7faccc13c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:19.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 1 -- 172.21.15.19:0/1317702472 shutdown_connections 2024-03-25T19:29:19.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 1 -- 172.21.15.19:0/1317702472 wait complete. 2024-03-25T19:29:19.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:19.365+0000 7facd27c8640 1 librados: shutdown 2024-03-25T19:29:19.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:23.345 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:22.300 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:23.349 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:22.304 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:23.462 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:22.448 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:23.466 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:22.452 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:23.583 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:22.559 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:23.587 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:22.564 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:23.591 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:22.554 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:23.595 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:22.558 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:23.784 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:22.740 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:23.788 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:22.743 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:23.914 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:22.856 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:23.918 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:22.860 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:24.415 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:23.388 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:24.419 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:23.392 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:24.544 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:23.532 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:24.548 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:23.536 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:26.185 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:26.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.319+0000 7f07a6da8640 10 monclient: get_monmap_and_config 2024-03-25T19:29:26.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.320+0000 7f07a6da8640 10 monclient: build_initial_monmap 2024-03-25T19:29:26.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.320+0000 7f07a6da8640 10 monclient: monmap: 2024-03-25T19:29:26.320 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:26.320 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:26.320 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:29:26.321253+0000 2024-03-25T19:29:26.320 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:29:26.321253+0000 2024-03-25T19:29:26.320 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:26.320 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:26.321 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:26.321 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:29:26.321 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:29:26.321 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.320+0000 7f07a6da8640 1 Processor -- start 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.321+0000 7f07a6da8640 1 -- start start 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.321+0000 7f07a6da8640 10 monclient: init 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.321+0000 7f07a6da8640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.321+0000 7f07a6da8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.321+0000 7f07a6da8640 10 monclient(hunting): picked mon.noname-b con 0x7f07a00b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.321+0000 7f07a6da8640 10 monclient(hunting): picked mon.noname-a con 0x7f07a00b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.322+0000 7f07a6da8640 10 monclient(hunting): picked mon.noname-c con 0x7f07a014be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.322+0000 7f07a6da8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a00af940 con 0x7f07a00b1730 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.322+0000 7f07a6da8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a00b0a70 con 0x7f07a014be90 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.322+0000 7f07a6da8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a00b0d50 con 0x7f07a00b4d60 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.322+0000 7f07a6da8640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.322+0000 7f07a6da8640 10 monclient(hunting): authenticate will time out at 6019.592409s 2024-03-25T19:29:26.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.322+0000 7f07a4b44640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f07a00b4d60 0x7f07a013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:50776/0 (socket says 172.21.15.19:50776) 2024-03-25T19:29:26.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.323+0000 7f07a4b44640 1 -- 172.21.15.19:0/821809076 learned_addr learned my addr 172.21.15.19:0/821809076 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:26.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.326+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712147154 0 0) 0x7f07a00b0a70 con 0x7f07a014be90 2024-03-25T19:29:26.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.326+0000 7f079f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:26.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.326+0000 7f079f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:26.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.326+0000 7f079f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:26.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.326+0000 7f079f7fe640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:29:26.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.326+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0790003520 con 0x7f07a014be90 2024-03-25T19:29:26.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.327+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111123428 0 0) 0x7f07a00b0d50 con 0x7f07a00b4d60 2024-03-25T19:29:26.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.327+0000 7f079f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:26.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.327+0000 7f079f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:26.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.327+0000 7f079f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:26.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.327+0000 7f079f7fe640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:29:26.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.327+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a00b0a70 con 0x7f07a00b4d60 2024-03-25T19:29:26.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.327+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104581775 0 0) 0x7f07a00af940 con 0x7f07a00b1730 2024-03-25T19:29:26.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.327+0000 7f079f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:26.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.327+0000 7f079f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:26.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.327+0000 7f079f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:26.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.327+0000 7f079f7fe640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:29:26.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.327+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a00b0d50 con 0x7f07a00b1730 2024-03-25T19:29:26.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1725182897 0 0) 0x7f0790003520 con 0x7f07a014be90 2024-03-25T19:29:26.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 >> v1:172.21.15.19:6790/0 conn(0x7f07a00b4d60 legacy=0x7f07a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:26.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 >> v1:172.21.15.19:6789/0 conn(0x7f07a00b1730 legacy=0x7f07a00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:26.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f079f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:29:26.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f079f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:29:26.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f079f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f079f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a00b1030 con 0x7f07a014be90 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f079f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f079f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f079f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f07a6da8640 5 monclient: authenticate success, global_id 4216 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.331+0000 7f07a6da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.332+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f07980019d0 con 0x7f07a014be90 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.332+0000 7f079f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.332+0000 7f079f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.332+0000 7f079f7fe640 10 monclient: dump: 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:26.333 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:26.334 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:26.334 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:26.334 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:26.334 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:26.334 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:26.334 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:26.334 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:26.334 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:26.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.332+0000 7f07a6da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:26.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.333+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0798001c60 con 0x7f07a014be90 2024-03-25T19:29:26.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.333+0000 7f079f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:26.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.333+0000 7f079f7fe640 1 -- 172.21.15.19:0/821809076 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0798001ff0 con 0x7f07a014be90 2024-03-25T19:29:26.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.333+0000 7f079f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:26.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.333+0000 7f079f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.333+0000 7f079f7fe640 10 monclient: dump: 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.333+0000 7f07a6da8640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.333+0000 7f07a6da8640 4 set_mon_vals no callback set 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.334+0000 7f07a6da8640 10 monclient: shutdown 2024-03-25T19:29:26.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.334+0000 7f07a6da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.334+0000 7f07a6da8640 1 -- 172.21.15.19:0/821809076 >> v1:172.21.15.188:6789/0 conn(0x7f07a014be90 legacy=0x7f07a014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.334+0000 7f07a6da8640 1 -- 172.21.15.19:0/821809076 shutdown_connections 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.334+0000 7f07a6da8640 1 -- 172.21.15.19:0/821809076 >> 172.21.15.19:0/821809076 conn(0x7f07a00617a0 msgr2=0x7f07a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.334+0000 7f07a6da8640 1 -- 172.21.15.19:0/821809076 shutdown_connections 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.334+0000 7f07a6da8640 1 -- 172.21.15.19:0/821809076 wait complete. 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.335+0000 7f07a6da8640 10 monclient: build_initial_monmap 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.335+0000 7f07a6da8640 10 monclient: monmap: 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:26.338 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:29:26.339 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:29:26.339 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:26.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.336+0000 7f07a6da8640 1 librados: starting msgr at 2024-03-25T19:29:26.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.336+0000 7f07a6da8640 1 librados: starting objecter 2024-03-25T19:29:26.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.336+0000 7f07a6da8640 1 Processor -- start 2024-03-25T19:29:26.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.336+0000 7f07a6da8640 1 -- start start 2024-03-25T19:29:26.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.336+0000 7f07a6da8640 1 librados: setting wanted keys 2024-03-25T19:29:26.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.336+0000 7f07a6da8640 1 librados: calling monclient init 2024-03-25T19:29:26.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.336+0000 7f07a6da8640 10 monclient: init 2024-03-25T19:29:26.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.337+0000 7f07a6da8640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:26.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.337+0000 7f07a6da8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:26.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.337+0000 7f07a6da8640 10 monclient(hunting): picked mon.noname-a con 0x7f07a00b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:26.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.337+0000 7f07a6da8640 10 monclient(hunting): picked mon.noname-c con 0x7f07a00b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:26.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.337+0000 7f07a6da8640 10 monclient(hunting): picked mon.noname-b con 0x7f07a014be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:26.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.337+0000 7f07a6da8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a01d75c0 con 0x7f07a00b1730 2024-03-25T19:29:26.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.337+0000 7f07a6da8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a013d0f0 con 0x7f07a014be90 2024-03-25T19:29:26.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.337+0000 7f07a6da8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a013d280 con 0x7f07a00b4d60 2024-03-25T19:29:26.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.337+0000 7f07a6da8640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:26.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.337+0000 7f07a6da8640 10 monclient(hunting): authenticate will time out at 6019.607889s 2024-03-25T19:29:26.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.338+0000 7f07a4b44640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f07a00b1730 0x7f07a01d7aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:58762/0 (socket says 172.21.15.19:58762) 2024-03-25T19:29:26.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.338+0000 7f07a4b44640 1 -- 172.21.15.19:0/3226520550 learned_addr learned my addr 172.21.15.19:0/3226520550 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:26.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.338+0000 7f079d7fa640 10 client.?.objecter ms_handle_connect 0x7f07a00b1730 2024-03-25T19:29:26.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.338+0000 7f079d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:26.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.338+0000 7f079d7fa640 10 client.?.objecter ms_handle_connect 0x7f07a014be90 2024-03-25T19:29:26.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.338+0000 7f079d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:26.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.339+0000 7f079d7fa640 10 client.?.objecter ms_handle_connect 0x7f07a00b4d60 2024-03-25T19:29:26.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.339+0000 7f079d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:26.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.341+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231190610 0 0) 0x7f07a013d0f0 con 0x7f07a014be90 2024-03-25T19:29:26.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.341+0000 7f079d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:26.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.341+0000 7f079d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:26.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.341+0000 7f079d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:26.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.341+0000 7f079d7fa640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:29:26.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.341+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0784004eb0 con 0x7f07a014be90 2024-03-25T19:29:26.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.341+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275812830 0 0) 0x7f078c001070 con 0x7f07a00b4d60 2024-03-25T19:29:26.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.341+0000 7f079d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:26.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.341+0000 7f079d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:26.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.341+0000 7f079d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:26.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.341+0000 7f079d7fa640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:29:26.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.341+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a013d0f0 con 0x7f07a00b4d60 2024-03-25T19:29:26.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.342+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345302635 0 0) 0x7f07a01d75c0 con 0x7f07a00b1730 2024-03-25T19:29:26.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.342+0000 7f079d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:26.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.342+0000 7f079d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:26.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.342+0000 7f079d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:26.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.342+0000 7f079d7fa640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:29:26.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.342+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f078c001070 con 0x7f07a00b1730 2024-03-25T19:29:26.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.348+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3684214310 0 0) 0x7f0784004eb0 con 0x7f07a014be90 2024-03-25T19:29:26.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.348+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07a01d75c0 con 0x7f07a014be90 2024-03-25T19:29:26.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.348+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0798001f70 con 0x7f07a014be90 2024-03-25T19:29:26.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.348+0000 7f079d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:26.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.348+0000 7f079d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:26.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.348+0000 7f079d7fa640 10 monclient(hunting): dump: 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.348+0000 7f079d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.348+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 192937616 0 0) 0x7f078c001070 con 0x7f07a00b1730 2024-03-25T19:29:26.349 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.348+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0784004eb0 con 0x7f07a00b1730 2024-03-25T19:29:26.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.349+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1200677521 0 0) 0x7f07a013d0f0 con 0x7f07a00b4d60 2024-03-25T19:29:26.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.349+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f078c001070 con 0x7f07a00b4d60 2024-03-25T19:29:26.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.349+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f078c001890 con 0x7f07a00b4d60 2024-03-25T19:29:26.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.349+0000 7f079d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:26.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.349+0000 7f079d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:26.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.349+0000 7f079d7fa640 10 monclient(hunting): dump: 2024-03-25T19:29:26.350 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:26.350 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:26.350 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:26.350 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:26.351 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:26.351 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:26.351 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:26.351 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:26.351 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:26.351 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:26.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2652356852 0 0) 0x7f07a01d75c0 con 0x7f07a014be90 2024-03-25T19:29:26.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 >> v1:172.21.15.19:6790/0 conn(0x7f07a00b4d60 legacy=0x7f07a0139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:26.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 >> v1:172.21.15.19:6789/0 conn(0x7f07a00b1730 legacy=0x7f07a01d7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:26.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f079d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:29:26.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f079d7fa640 1 monclient: found mon.b 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f079d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f079d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a0139740 con 0x7f07a014be90 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f079d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f079d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f079d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f07a6da8640 5 monclient: authenticate success, global_id 4219 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f07a6da8640 10 monclient: _renew_subs 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.353+0000 7f07a6da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.354+0000 7f07a6da8640 1 -- 172.21.15.19:0/3226520550 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a01d81b0 con 0x7f07a014be90 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.354+0000 7f07a6da8640 10 client.4219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.354+0000 7f07a6da8640 10 monclient: _renew_subs 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.354+0000 7f07a6da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:26.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.354+0000 7f07a6da8640 1 -- 172.21.15.19:0/3226520550 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a013a6a0 con 0x7f07a014be90 2024-03-25T19:29:26.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.354+0000 7f07a6da8640 1 librados: init done 2024-03-25T19:29:26.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.355+0000 7f07a6da8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:26.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.355+0000 7f07a6da8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:26.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.355+0000 7f07a6da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:26.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.355+0000 7f07a6da8640 1 -- 172.21.15.19:0/3226520550 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a01d77c0 con 0x7f07a014be90 2024-03-25T19:29:26.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.381+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07980020d0 con 0x7f07a014be90 2024-03-25T19:29:26.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.381+0000 7f079d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:26.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.381+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0798003c20 con 0x7f07a014be90 2024-03-25T19:29:26.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.381+0000 7f079d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:26.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.381+0000 7f079d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:26.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.381+0000 7f079d7fa640 10 monclient: dump: 2024-03-25T19:29:26.381 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:26.381 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:26.381 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:26.381 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:26.382 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:26.382 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:26.382 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:26.382 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:26.382 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:26.382 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:26.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.381+0000 7f079effd640 4 set_mon_vals no callback set 2024-03-25T19:29:26.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.382+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f0798019bf0 con 0x7f07a014be90 2024-03-25T19:29:26.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.383+0000 7f07a4b44640 10 monclient: get_auth_request con 0x7f078406e270 auth_method 0 2024-03-25T19:29:26.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.383+0000 7f079d7fa640 10 client.4219.objecter ms_handle_connect 0x7f078406e270 2024-03-25T19:29:26.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.386+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f07980536d0 con 0x7f07a014be90 2024-03-25T19:29:26.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.386+0000 7f079d7fa640 10 client.4219.objecter ms_dispatch 0x7f07a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:26.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.386+0000 7f079d7fa640 3 client.4219.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:26.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.386+0000 7f079d7fa640 3 client.4219.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:26.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.386+0000 7f079d7fa640 20 client.4219.objecter dump_active .. 0 homeless 2024-03-25T19:29:26.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.694+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f079804f1e0 con 0x7f07a014be90 2024-03-25T19:29:26.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.694+0000 7f079d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:26.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.694+0000 7f079d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:26.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.924+0000 7f07a6da8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:26.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.924+0000 7f07a6da8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:26.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.924+0000 7f07a6da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:26.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.924+0000 7f07a6da8640 1 -- 172.21.15.19:0/3226520550 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07a00627b0 con 0x7f07a014be90 2024-03-25T19:29:26.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.950+0000 7f079d7fa640 1 -- 172.21.15.19:0/3226520550 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f0798051110 con 0x7f07a014be90 2024-03-25T19:29:26.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.950+0000 7f079d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:26.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.950+0000 7f079d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:26.950 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:29:26.950 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:26.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.952+0000 7f07a6da8640 10 librados: watch_flush enter 2024-03-25T19:29:26.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.952+0000 7f07a6da8640 10 librados: watch_flush exit 2024-03-25T19:29:26.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.952+0000 7f07a6da8640 20 client.4219.objecter shutdown clearing up homeless session... 2024-03-25T19:29:26.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.952+0000 7f07a6da8640 10 client.4219.objecter successfully canceled tick 2024-03-25T19:29:26.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.953+0000 7f07a6da8640 1 -- 172.21.15.19:0/3226520550 >> v1:172.21.15.188:6800/2207506229 conn(0x7f078406e270 legacy=0x7f0784070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:26.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.953+0000 7f07a6da8640 10 monclient: shutdown 2024-03-25T19:29:26.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.953+0000 7f07a6da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:26.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.953+0000 7f07a6da8640 1 -- 172.21.15.19:0/3226520550 >> v1:172.21.15.188:6789/0 conn(0x7f07a014be90 legacy=0x7f07a01e20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:26.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.953+0000 7f07a6da8640 1 -- 172.21.15.19:0/3226520550 shutdown_connections 2024-03-25T19:29:26.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.953+0000 7f07a6da8640 1 -- 172.21.15.19:0/3226520550 >> 172.21.15.19:0/3226520550 conn(0x7f07a00617a0 msgr2=0x7f07a00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:26.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.953+0000 7f07a6da8640 1 -- 172.21.15.19:0/3226520550 shutdown_connections 2024-03-25T19:29:26.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.953+0000 7f07a6da8640 1 -- 172.21.15.19:0/3226520550 wait complete. 2024-03-25T19:29:26.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:26.953+0000 7f07a6da8640 1 librados: shutdown 2024-03-25T19:29:26.966 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:27.415 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:00:26.378 35606== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:27.519 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:26.505 36133== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:27.716 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:00:26.671 35613== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:27.854 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:26.830 36135== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:27.970 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:00:26.912 35615== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:28.276 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:00:27.232 35614== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:28.890 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:27.864 36137== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:28.954 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:27.942 36131== Warning: unimplemented fcntl command: 1038 2024-03-25T19:29:33.774 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:33.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.872+0000 7f14c21c6640 10 monclient: get_monmap_and_config 2024-03-25T19:29:33.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.872+0000 7f14c21c6640 10 monclient: build_initial_monmap 2024-03-25T19:29:33.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.872+0000 7f14c21c6640 10 monclient: monmap: 2024-03-25T19:29:33.872 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:33.872 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:33.873 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:29:33.873421+0000 2024-03-25T19:29:33.873 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:29:33.873421+0000 2024-03-25T19:29:33.873 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:33.873 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:33.873 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:33.873 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:29:33.873 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:29:33.873 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:33.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.873+0000 7f14c21c6640 1 Processor -- start 2024-03-25T19:29:33.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.873+0000 7f14c21c6640 1 -- start start 2024-03-25T19:29:33.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.873+0000 7f14c21c6640 10 monclient: init 2024-03-25T19:29:33.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.874+0000 7f14c21c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:33.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.874+0000 7f14c21c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:33.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.874+0000 7f14c21c6640 10 monclient(hunting): picked mon.noname-b con 0x7f14bc0b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:33.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.874+0000 7f14c21c6640 10 monclient(hunting): picked mon.noname-c con 0x7f14bc0b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:33.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.874+0000 7f14c21c6640 10 monclient(hunting): picked mon.noname-a con 0x7f14bc14be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:33.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.874+0000 7f14c21c6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14bc0af940 con 0x7f14bc14be90 2024-03-25T19:29:33.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.874+0000 7f14c21c6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14bc0b0a70 con 0x7f14bc0b1730 2024-03-25T19:29:33.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.874+0000 7f14c21c6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14bc0b0d50 con 0x7f14bc0b4d60 2024-03-25T19:29:33.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.874+0000 7f14c21c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:33.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.874+0000 7f14c21c6640 10 monclient(hunting): authenticate will time out at 6027.144299s 2024-03-25T19:29:33.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.905+0000 7f14bb7fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f14bc0b4d60 0x7f14bc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:33430/0 (socket says 172.21.15.19:33430) 2024-03-25T19:29:33.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.905+0000 7f14bb7fe640 1 -- 172.21.15.19:0/1575857662 learned_addr learned my addr 172.21.15.19:0/1575857662 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:33.933 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.932+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274032731 0 0) 0x7f14bc0b0d50 con 0x7f14bc0b4d60 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.932+0000 7f14ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.932+0000 7f14ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.932+0000 7f14ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.932+0000 7f14ba7fc640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.932+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a4003520 con 0x7f14bc0b4d60 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.932+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382500119 0 0) 0x7f14bc0b0a70 con 0x7f14bc0b1730 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.932+0000 7f14ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.932+0000 7f14ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.932+0000 7f14ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.932+0000 7f14ba7fc640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.932+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14bc0b0d50 con 0x7f14bc0b1730 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.933+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744801006 0 0) 0x7f14bc0af940 con 0x7f14bc14be90 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.933+0000 7f14ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.933+0000 7f14ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.933+0000 7f14ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.933+0000 7f14ba7fc640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:29:33.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.933+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14bc0b0a70 con 0x7f14bc14be90 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1416493462 0 0) 0x7f14a4003520 con 0x7f14bc0b4d60 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 >> v1:172.21.15.188:6789/0 conn(0x7f14bc0b1730 legacy=0x7f14bc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 >> v1:172.21.15.19:6789/0 conn(0x7f14bc14be90 legacy=0x7f14bc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 1 monclient: found mon.noname-b 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc0b1030 con 0x7f14bc0b4d60 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 10 monclient: _finish_auth 0 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14c21c6640 5 monclient: authenticate success, global_id 4211 2024-03-25T19:29:33.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14c21c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:33.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f14ac001650 con 0x7f14bc0b4d60 2024-03-25T19:29:33.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:33.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14ba7fc640 10 monclient: dump: 2024-03-25T19:29:33.938 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:33.938 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:33.938 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:33.939 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:33.939 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.939 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:33.939 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:33.939 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:33.939 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:33.939 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:33.939 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.937+0000 7f14c21c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.939+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14ac0018a0 con 0x7f14bc0b4d60 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.939+0000 7f14ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.939+0000 7f14ba7fc640 1 -- 172.21.15.19:0/1575857662 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f14ac001c30 con 0x7f14bc0b4d60 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.939+0000 7f14ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.939+0000 7f14ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.939+0000 7f14ba7fc640 10 monclient: dump: 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:33.940 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:33.941 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:33.941 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:33.941 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:33.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.939+0000 7f14c21c6640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:33.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.939+0000 7f14c21c6640 4 set_mon_vals no callback set 2024-03-25T19:29:33.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.940+0000 7f14c21c6640 10 monclient: shutdown 2024-03-25T19:29:33.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.940+0000 7f14c21c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:33.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.940+0000 7f14c21c6640 1 -- 172.21.15.19:0/1575857662 >> v1:172.21.15.19:6790/0 conn(0x7f14bc0b4d60 legacy=0x7f14bc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.940+0000 7f14c21c6640 1 -- 172.21.15.19:0/1575857662 shutdown_connections 2024-03-25T19:29:33.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.940+0000 7f14c21c6640 1 -- 172.21.15.19:0/1575857662 >> 172.21.15.19:0/1575857662 conn(0x7f14bc0617a0 msgr2=0x7f14bc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:33.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.941+0000 7f14c21c6640 1 -- 172.21.15.19:0/1575857662 shutdown_connections 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.941+0000 7f14c21c6640 1 -- 172.21.15.19:0/1575857662 wait complete. 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.942+0000 7f14c21c6640 10 monclient: build_initial_monmap 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.942+0000 7f14c21c6640 10 monclient: monmap: 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.942+0000 7f14c21c6640 1 librados: starting msgr at 2024-03-25T19:29:33.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.942+0000 7f14c21c6640 1 librados: starting objecter 2024-03-25T19:29:33.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.942+0000 7f14c21c6640 1 Processor -- start 2024-03-25T19:29:33.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.942+0000 7f14c21c6640 1 -- start start 2024-03-25T19:29:33.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.942+0000 7f14c21c6640 1 librados: setting wanted keys 2024-03-25T19:29:33.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.942+0000 7f14c21c6640 1 librados: calling monclient init 2024-03-25T19:29:33.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.942+0000 7f14c21c6640 10 monclient: init 2024-03-25T19:29:33.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.943+0000 7f14c21c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:33.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.943+0000 7f14c21c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:33.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.943+0000 7f14c21c6640 10 monclient(hunting): picked mon.noname-a con 0x7f14bc0b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:33.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.943+0000 7f14c21c6640 10 monclient(hunting): picked mon.noname-c con 0x7f14bc0b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:33.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.943+0000 7f14c21c6640 10 monclient(hunting): picked mon.noname-b con 0x7f14bc14be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:33.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.943+0000 7f14c21c6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14bc137760 con 0x7f14bc0b1730 2024-03-25T19:29:33.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.943+0000 7f14c21c6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14bc137280 con 0x7f14bc14be90 2024-03-25T19:29:33.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.943+0000 7f14c21c6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14bc137410 con 0x7f14bc0b4d60 2024-03-25T19:29:33.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.943+0000 7f14c21c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:33.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.943+0000 7f14c21c6640 10 monclient(hunting): authenticate will time out at 6027.213661s 2024-03-25T19:29:33.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.944+0000 7f14baffd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f14bc0b4d60 0x7f14bc12fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:33444/0 (socket says 172.21.15.19:33444) 2024-03-25T19:29:33.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.944+0000 7f14baffd640 1 -- 172.21.15.19:0/1970984396 learned_addr learned my addr 172.21.15.19:0/1970984396 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:33.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.944+0000 7f14a3fff640 10 client.?.objecter ms_handle_connect 0x7f14bc0b4d60 2024-03-25T19:29:33.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.944+0000 7f14a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:33.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.945+0000 7f14a3fff640 10 client.?.objecter ms_handle_connect 0x7f14bc14be90 2024-03-25T19:29:33.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.945+0000 7f14a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:33.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.945+0000 7f14a3fff640 10 client.?.objecter ms_handle_connect 0x7f14bc0b1730 2024-03-25T19:29:33.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.945+0000 7f14a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:33.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.947+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745129829 0 0) 0x7f14bc137280 con 0x7f14bc14be90 2024-03-25T19:29:33.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.947+0000 7f14a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.947+0000 7f14a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.947+0000 7f14a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.947+0000 7f14a3fff640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:29:33.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.947+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a8004eb0 con 0x7f14bc14be90 2024-03-25T19:29:33.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.948+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561543738 0 0) 0x7f14bc137760 con 0x7f14bc0b1730 2024-03-25T19:29:33.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.948+0000 7f14a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.948+0000 7f14a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.948+0000 7f14a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.948+0000 7f14a3fff640 10 monclient(hunting): my global_id is 4227 2024-03-25T19:29:33.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.948+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14bc137280 con 0x7f14bc0b1730 2024-03-25T19:29:33.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.948+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404715854 0 0) 0x7f14bc137410 con 0x7f14bc0b4d60 2024-03-25T19:29:33.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.948+0000 7f14a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.948+0000 7f14a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.948+0000 7f14a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.948+0000 7f14a3fff640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:29:33.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.948+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14bc137760 con 0x7f14bc0b4d60 2024-03-25T19:29:33.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.953+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1546579789 0 0) 0x7f14bc137280 con 0x7f14bc0b1730 2024-03-25T19:29:33.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.953+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14bc137410 con 0x7f14bc0b1730 2024-03-25T19:29:33.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.953+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f14ac001bb0 con 0x7f14bc0b1730 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.953+0000 7f14a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.953+0000 7f14a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.953+0000 7f14a3fff640 10 monclient(hunting): dump: 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:33.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.953+0000 7f14a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:33.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.955+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3508587853 0 0) 0x7f14a8004eb0 con 0x7f14bc14be90 2024-03-25T19:29:33.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.955+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14bc137280 con 0x7f14bc14be90 2024-03-25T19:29:33.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.955+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f14b4001bd0 con 0x7f14bc14be90 2024-03-25T19:29:33.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.955+0000 7f14a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.955+0000 7f14a3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:33.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.955+0000 7f14a3fff640 10 monclient(hunting): dump: 2024-03-25T19:29:33.955 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:33.955 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:33.955 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:33.955 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:33.956 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.956 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:33.956 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:33.956 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:33.956 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:33.956 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:33.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.956+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 538930048 0 0) 0x7f14bc137760 con 0x7f14bc0b4d60 2024-03-25T19:29:33.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.956+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14a8004eb0 con 0x7f14bc0b4d60 2024-03-25T19:29:33.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.956+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f14b0001930 con 0x7f14bc0b4d60 2024-03-25T19:29:33.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.956+0000 7f14a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.956+0000 7f14a3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:33.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.956+0000 7f14a3fff640 10 monclient(hunting): dump: 2024-03-25T19:29:33.957 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:33.957 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 226684872 0 0) 0x7f14bc137410 con 0x7f14bc0b1730 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 >> v1:172.21.15.19:6790/0 conn(0x7f14bc0b4d60 legacy=0x7f14bc12fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 >> v1:172.21.15.188:6789/0 conn(0x7f14bc14be90 legacy=0x7f14bc1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14a3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14a3fff640 1 monclient: found mon.a 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:33.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:33.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14bc130570 con 0x7f14bc0b1730 2024-03-25T19:29:33.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14a3fff640 10 monclient: _finish_auth 0 2024-03-25T19:29:33.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14a3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:33.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:33.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14c21c6640 5 monclient: authenticate success, global_id 4227 2024-03-25T19:29:33.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14c21c6640 10 monclient: _renew_subs 2024-03-25T19:29:33.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14c21c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:33.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14c21c6640 1 -- 172.21.15.19:0/1970984396 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14bc1307c0 con 0x7f14bc0b1730 2024-03-25T19:29:33.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14c21c6640 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:33.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14c21c6640 10 monclient: _renew_subs 2024-03-25T19:29:33.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14c21c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:33.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14c21c6640 1 -- 172.21.15.19:0/1970984396 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14bc13e690 con 0x7f14bc0b1730 2024-03-25T19:29:33.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.957+0000 7f14c21c6640 1 librados: init done 2024-03-25T19:29:33.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.958+0000 7f14c21c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:33.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.958+0000 7f14c21c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:33.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.958+0000 7f14c21c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:33.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:33.958+0000 7f14c21c6640 1 -- 172.21.15.19:0/1970984396 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14bc1375a0 con 0x7f14bc0b1730 2024-03-25T19:29:34.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.251+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14ac001d10 con 0x7f14bc0b1730 2024-03-25T19:29:34.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.251+0000 7f14a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:34.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.251+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f14ac004d90 con 0x7f14bc0b1730 2024-03-25T19:29:34.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.251+0000 7f14a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:34.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.251+0000 7f14a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:34.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.251+0000 7f14a3fff640 10 monclient: dump: 2024-03-25T19:29:34.251 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:34.251 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:34.251 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:34.252 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:34.252 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:34.252 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:34.252 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:34.252 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:34.252 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:34.252 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:34.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.251+0000 7f14b9ffb640 4 set_mon_vals no callback set 2024-03-25T19:29:34.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.252+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f14ac01ad60 con 0x7f14bc0b1730 2024-03-25T19:29:34.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.253+0000 7f14baffd640 10 monclient: get_auth_request con 0x7f14a806e0f0 auth_method 0 2024-03-25T19:29:34.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.253+0000 7f14a3fff640 10 client.4227.objecter ms_handle_connect 0x7f14a806e0f0 2024-03-25T19:29:34.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.258+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f14ac054850 con 0x7f14bc0b1730 2024-03-25T19:29:34.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.258+0000 7f14a3fff640 10 client.4227.objecter ms_dispatch 0x7f14bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:34.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.258+0000 7f14a3fff640 3 client.4227.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:34.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.258+0000 7f14a3fff640 3 client.4227.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:34.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.258+0000 7f14a3fff640 20 client.4227.objecter dump_active .. 0 homeless 2024-03-25T19:29:34.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.258+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14ac050190 con 0x7f14bc0b1730 2024-03-25T19:29:34.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.258+0000 7f14a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:34.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.258+0000 7f14a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:34.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.484+0000 7f14c21c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:34.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.484+0000 7f14c21c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:34.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.484+0000 7f14c21c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:34.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.484+0000 7f14c21c6640 1 -- 172.21.15.19:0/1970984396 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14bc0627b0 con 0x7f14bc0b1730 2024-03-25T19:29:34.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.511+0000 7f14a3fff640 1 -- 172.21.15.19:0/1970984396 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f14ac052290 con 0x7f14bc0b1730 2024-03-25T19:29:34.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.512+0000 7f14a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:34.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.512+0000 7f14a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:34.512 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:29:34.512 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.514+0000 7f14c21c6640 10 librados: watch_flush enter 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.514+0000 7f14c21c6640 10 librados: watch_flush exit 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.514+0000 7f14c21c6640 20 client.4227.objecter shutdown clearing up homeless session... 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.514+0000 7f14c21c6640 10 client.4227.objecter successfully canceled tick 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.514+0000 7f14c21c6640 1 -- 172.21.15.19:0/1970984396 >> v1:172.21.15.188:6800/2207506229 conn(0x7f14a806e0f0 legacy=0x7f14a80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.514+0000 7f14c21c6640 10 monclient: shutdown 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.514+0000 7f14c21c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.514+0000 7f14c21c6640 1 -- 172.21.15.19:0/1970984396 >> v1:172.21.15.19:6789/0 conn(0x7f14bc0b1730 legacy=0x7f14bc143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.514+0000 7f14c21c6640 1 -- 172.21.15.19:0/1970984396 shutdown_connections 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.514+0000 7f14c21c6640 1 -- 172.21.15.19:0/1970984396 >> 172.21.15.19:0/1970984396 conn(0x7f14bc0617a0 msgr2=0x7f14bc137f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.515+0000 7f14c21c6640 1 -- 172.21.15.19:0/1970984396 shutdown_connections 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.515+0000 7f14c21c6640 1 -- 172.21.15.19:0/1970984396 wait complete. 2024-03-25T19:29:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:34.515+0000 7f14c21c6640 1 librados: shutdown 2024-03-25T19:29:34.530 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:35.813 INFO:tasks.ceph.osd.4.smithi188.stderr:2024-03-25T19:29:35.806+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:29:35.826 INFO:tasks.ceph.osd.5.smithi188.stderr:2024-03-25T19:29:35.823+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:29:36.084 INFO:tasks.ceph.osd.7.smithi188.stderr:2024-03-25T19:29:36.082+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:29:36.170 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-03-25T19:29:36.167+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:29:36.784 INFO:tasks.ceph.osd.6.smithi188.stderr:2024-03-25T19:29:36.781+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:29:36.949 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-03-25T19:29:36.891+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:29:38.532 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-03-25T19:29:38.528+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:29:38.773 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-03-25T19:29:38.770+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:29:41.337 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:41.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.454+0000 7f20eb94b640 10 monclient: get_monmap_and_config 2024-03-25T19:29:41.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.454+0000 7f20eb94b640 10 monclient: build_initial_monmap 2024-03-25T19:29:41.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.454+0000 7f20eb94b640 10 monclient: monmap: 2024-03-25T19:29:41.454 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:41.454 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:41.454 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:29:41.455709+0000 2024-03-25T19:29:41.455 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:29:41.455709+0000 2024-03-25T19:29:41.455 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:41.455 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:41.455 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:41.455 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:29:41.455 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:29:41.455 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.455+0000 7f20eb94b640 1 Processor -- start 2024-03-25T19:29:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.455+0000 7f20eb94b640 1 -- start start 2024-03-25T19:29:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.455+0000 7f20eb94b640 10 monclient: init 2024-03-25T19:29:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.456+0000 7f20eb94b640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.456+0000 7f20eb94b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:41.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.456+0000 7f20eb94b640 10 monclient(hunting): picked mon.noname-a con 0x7f20e40a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:41.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.456+0000 7f20eb94b640 10 monclient(hunting): picked mon.noname-b con 0x7f20e413f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:41.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.456+0000 7f20eb94b640 10 monclient(hunting): picked mon.noname-c con 0x7f20e41430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:41.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.456+0000 7f20eb94b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e4142520 con 0x7f20e40a98f0 2024-03-25T19:29:41.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.456+0000 7f20eb94b640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e40b28a0 con 0x7f20e41430c0 2024-03-25T19:29:41.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.456+0000 7f20eb94b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e40b2a30 con 0x7f20e413f710 2024-03-25T19:29:41.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.456+0000 7f20eb94b640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:41.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.456+0000 7f20eb94b640 10 monclient(hunting): authenticate will time out at 6034.726389s 2024-03-25T19:29:41.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.467+0000 7f20e8ee6640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f20e413f710 0x7f20e41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:55534/0 (socket says 172.21.15.19:55534) 2024-03-25T19:29:41.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.468+0000 7f20e8ee6640 1 -- 172.21.15.19:0/477901799 learned_addr learned my addr 172.21.15.19:0/477901799 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:41.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.478+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071142043 0 0) 0x7f20e40b28a0 con 0x7f20e41430c0 2024-03-25T19:29:41.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.478+0000 7f20dbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:41.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.478+0000 7f20dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:41.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.478+0000 7f20dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:41.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.478+0000 7f20dbfff640 10 monclient(hunting): my global_id is 4249 2024-03-25T19:29:41.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.478+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20d0003520 con 0x7f20e41430c0 2024-03-25T19:29:41.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.479+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087613559 0 0) 0x7f20e40b2a30 con 0x7f20e413f710 2024-03-25T19:29:41.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.479+0000 7f20dbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:41.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.479+0000 7f20dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:41.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.479+0000 7f20dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:41.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.479+0000 7f20dbfff640 10 monclient(hunting): my global_id is 4238 2024-03-25T19:29:41.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.479+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e40b28a0 con 0x7f20e413f710 2024-03-25T19:29:41.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.484+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216507232 0 0) 0x7f20e4142520 con 0x7f20e40a98f0 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.484+0000 7f20dbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.484+0000 7f20dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.484+0000 7f20dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.484+0000 7f20dbfff640 10 monclient(hunting): my global_id is 4254 2024-03-25T19:29:41.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.484+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e40b2a30 con 0x7f20e40a98f0 2024-03-25T19:29:41.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.486+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2531609024 0 0) 0x7f20d0003520 con 0x7f20e41430c0 2024-03-25T19:29:41.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.486+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 >> v1:172.21.15.19:6790/0 conn(0x7f20e413f710 legacy=0x7f20e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.486+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 >> v1:172.21.15.19:6789/0 conn(0x7f20e40a98f0 legacy=0x7f20e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.486+0000 7f20dbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:41.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.486+0000 7f20dbfff640 1 monclient: found mon.noname-c 2024-03-25T19:29:41.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.486+0000 7f20dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:41.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.486+0000 7f20dbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:29:41.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.486+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e40b44c0 con 0x7f20e41430c0 2024-03-25T19:29:41.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.486+0000 7f20dbfff640 10 monclient: _finish_auth 0 2024-03-25T19:29:41.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.487+0000 7f20dbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.487+0000 7f20dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.487+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f20dc001a10 con 0x7f20e41430c0 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.487+0000 7f20dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.487+0000 7f20dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.487+0000 7f20dbfff640 10 monclient: dump: 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:41.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.487+0000 7f20eb94b640 5 monclient: authenticate success, global_id 4249 2024-03-25T19:29:41.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.487+0000 7f20eb94b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:41.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.504+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20dc001bf0 con 0x7f20e41430c0 2024-03-25T19:29:41.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.504+0000 7f20dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:41.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.504+0000 7f20dbfff640 1 -- 172.21.15.19:0/477901799 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f20dc002000 con 0x7f20e41430c0 2024-03-25T19:29:41.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.504+0000 7f20dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:41.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.504+0000 7f20dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:41.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.504+0000 7f20dbfff640 10 monclient: dump: 2024-03-25T19:29:41.504 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:41.504 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:41.504 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:41.505 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:41.505 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:41.505 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:41.505 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:41.505 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:41.505 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:41.505 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:41.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.504+0000 7f20eb94b640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:41.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.504+0000 7f20eb94b640 4 set_mon_vals no callback set 2024-03-25T19:29:41.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.505+0000 7f20eb94b640 10 monclient: shutdown 2024-03-25T19:29:41.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.505+0000 7f20eb94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:41.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.505+0000 7f20eb94b640 1 -- 172.21.15.19:0/477901799 >> v1:172.21.15.188:6789/0 conn(0x7f20e41430c0 legacy=0x7f20e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.505+0000 7f20eb94b640 1 -- 172.21.15.19:0/477901799 shutdown_connections 2024-03-25T19:29:41.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.505+0000 7f20eb94b640 1 -- 172.21.15.19:0/477901799 >> 172.21.15.19:0/477901799 conn(0x7f20e40b7ea0 msgr2=0x7f20e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:41.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.505+0000 7f20eb94b640 1 -- 172.21.15.19:0/477901799 shutdown_connections 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.505+0000 7f20eb94b640 1 -- 172.21.15.19:0/477901799 wait complete. 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.506+0000 7f20eb94b640 10 monclient: build_initial_monmap 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.506+0000 7f20eb94b640 10 monclient: monmap: 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:41.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.506+0000 7f20eb94b640 1 librados: starting msgr at 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.506+0000 7f20eb94b640 1 librados: starting objecter 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.506+0000 7f20eb94b640 1 Processor -- start 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.506+0000 7f20eb94b640 1 -- start start 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.506+0000 7f20eb94b640 1 librados: setting wanted keys 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.506+0000 7f20eb94b640 1 librados: calling monclient init 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.506+0000 7f20eb94b640 10 monclient: init 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.507+0000 7f20eb94b640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.507+0000 7f20eb94b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.507+0000 7f20eb94b640 10 monclient(hunting): picked mon.noname-a con 0x7f20e40a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.507+0000 7f20eb94b640 10 monclient(hunting): picked mon.noname-c con 0x7f20e413f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.507+0000 7f20eb94b640 10 monclient(hunting): picked mon.noname-b con 0x7f20e41430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.507+0000 7f20eb94b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e4132ba0 con 0x7f20e40a98f0 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.507+0000 7f20eb94b640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e4138150 con 0x7f20e41430c0 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.507+0000 7f20eb94b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e4139510 con 0x7f20e413f710 2024-03-25T19:29:41.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.507+0000 7f20eb94b640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:41.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.507+0000 7f20eb94b640 10 monclient(hunting): authenticate will time out at 6034.777536s 2024-03-25T19:29:41.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.518+0000 7f20e8ee6640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f20e413f710 0x7f20e4135000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:55592/0 (socket says 172.21.15.19:55592) 2024-03-25T19:29:41.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.518+0000 7f20e8ee6640 1 -- 172.21.15.19:0/22619683 learned_addr learned my addr 172.21.15.19:0/22619683 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:41.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.523+0000 7f20d9ffb640 10 client.?.objecter ms_handle_connect 0x7f20e413f710 2024-03-25T19:29:41.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.523+0000 7f20d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:41.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.529+0000 7f20d9ffb640 10 client.?.objecter ms_handle_connect 0x7f20e41430c0 2024-03-25T19:29:41.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.529+0000 7f20d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:41.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.532+0000 7f20d9ffb640 10 client.?.objecter ms_handle_connect 0x7f20e40a98f0 2024-03-25T19:29:41.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.532+0000 7f20d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:41.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.539+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809032083 0 0) 0x7f20e4138150 con 0x7f20e41430c0 2024-03-25T19:29:41.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.539+0000 7f20d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:41.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.539+0000 7f20d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:41.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.539+0000 7f20d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:41.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.539+0000 7f20d9ffb640 10 monclient(hunting): my global_id is 4252 2024-03-25T19:29:41.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.539+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20c8004eb0 con 0x7f20e41430c0 2024-03-25T19:29:41.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.544+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499577359 0 0) 0x7f20e4139510 con 0x7f20e413f710 2024-03-25T19:29:41.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.544+0000 7f20d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:41.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.544+0000 7f20d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:41.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.544+0000 7f20d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:41.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.544+0000 7f20d9ffb640 10 monclient(hunting): my global_id is 4241 2024-03-25T19:29:41.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.544+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e4138150 con 0x7f20e413f710 2024-03-25T19:29:41.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.545+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506548953 0 0) 0x7f20e40b2a30 con 0x7f20e40a98f0 2024-03-25T19:29:41.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.545+0000 7f20d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:41.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.545+0000 7f20d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:41.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.545+0000 7f20d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:41.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.545+0000 7f20d9ffb640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:29:41.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.545+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e4139510 con 0x7f20e40a98f0 2024-03-25T19:29:41.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.559+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 749214222 0 0) 0x7f20c8004eb0 con 0x7f20e41430c0 2024-03-25T19:29:41.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.559+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20e40b2a30 con 0x7f20e41430c0 2024-03-25T19:29:41.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.559+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f20dc001f00 con 0x7f20e41430c0 2024-03-25T19:29:41.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.559+0000 7f20d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:41.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.559+0000 7f20d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:41.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.559+0000 7f20d9ffb640 10 monclient(hunting): dump: 2024-03-25T19:29:41.559 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:41.559 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:41.559 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:41.559 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:41.559 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:41.559 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:41.560 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:41.560 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:41.560 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:41.560 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:41.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.559+0000 7f20d9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:41.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 477549100 0 0) 0x7f20e40b2a30 con 0x7f20e41430c0 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 >> v1:172.21.15.19:6790/0 conn(0x7f20e413f710 legacy=0x7f20e4135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 >> v1:172.21.15.19:6789/0 conn(0x7f20e40a98f0 legacy=0x7f20e4137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20d9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20d9ffb640 1 monclient: found mon.b 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20d9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e41397f0 con 0x7f20e41430c0 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20d9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20d9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20eb94b640 5 monclient: authenticate success, global_id 4252 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20eb94b640 10 monclient: _renew_subs 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20eb94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:41.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20eb94b640 1 -- 172.21.15.19:0/22619683 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e4138600 con 0x7f20e41430c0 2024-03-25T19:29:41.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20eb94b640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:41.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20eb94b640 10 monclient: _renew_subs 2024-03-25T19:29:41.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20eb94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:41.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20eb94b640 1 -- 172.21.15.19:0/22619683 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e4135950 con 0x7f20e41430c0 2024-03-25T19:29:41.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.578+0000 7f20eb94b640 1 librados: init done 2024-03-25T19:29:41.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.580+0000 7f20eb94b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:41.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.580+0000 7f20eb94b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:41.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.580+0000 7f20eb94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:41.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.580+0000 7f20eb94b640 1 -- 172.21.15.19:0/22619683 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e40627b0 con 0x7f20e41430c0 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.851+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20dc0020e0 con 0x7f20e41430c0 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.851+0000 7f20d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.851+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f20dc003dc0 con 0x7f20e41430c0 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.851+0000 7f20d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.851+0000 7f20d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.851+0000 7f20d9ffb640 10 monclient: dump: 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:41.851 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:41.852 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:41.852 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:41.852 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:41.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.851+0000 7f20db7fe640 4 set_mon_vals no callback set 2024-03-25T19:29:41.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.852+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f20dc019d90 con 0x7f20e41430c0 2024-03-25T19:29:41.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.852+0000 7f20e96e7640 10 monclient: get_auth_request con 0x7f20c806dd50 auth_method 0 2024-03-25T19:29:41.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.853+0000 7f20d9ffb640 10 client.4252.objecter ms_handle_connect 0x7f20c806dd50 2024-03-25T19:29:41.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.854+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f20dc053970 con 0x7f20e41430c0 2024-03-25T19:29:41.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.854+0000 7f20d9ffb640 10 client.4252.objecter ms_dispatch 0x7f20e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:41.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.854+0000 7f20d9ffb640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:41.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.854+0000 7f20d9ffb640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:41.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.854+0000 7f20d9ffb640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-25T19:29:41.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.859+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20dc04f2b0 con 0x7f20e41430c0 2024-03-25T19:29:41.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.859+0000 7f20d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:41.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:41.859+0000 7f20d9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:42.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.116+0000 7f20eb94b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:42.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.116+0000 7f20eb94b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:42.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.116+0000 7f20eb94b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:42.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.116+0000 7f20eb94b640 1 -- 172.21.15.19:0/22619683 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20e413dcf0 con 0x7f20e41430c0 2024-03-25T19:29:42.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.139+0000 7f20d9ffb640 1 -- 172.21.15.19:0/22619683 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f20dc0513b0 con 0x7f20e41430c0 2024-03-25T19:29:42.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.139+0000 7f20d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:42.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.139+0000 7f20d9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:42.139 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:29:42.139 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:42.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 10 librados: watch_flush enter 2024-03-25T19:29:42.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 10 librados: watch_flush exit 2024-03-25T19:29:42.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-25T19:29:42.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 10 client.4252.objecter successfully canceled tick 2024-03-25T19:29:42.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 1 -- 172.21.15.19:0/22619683 >> v1:172.21.15.188:6800/2207506229 conn(0x7f20c806dd50 legacy=0x7f20c8070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:42.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 10 monclient: shutdown 2024-03-25T19:29:42.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:42.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 1 -- 172.21.15.19:0/22619683 >> v1:172.21.15.188:6789/0 conn(0x7f20e41430c0 legacy=0x7f20e41e27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:42.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 1 -- 172.21.15.19:0/22619683 shutdown_connections 2024-03-25T19:29:42.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 1 -- 172.21.15.19:0/22619683 >> 172.21.15.19:0/22619683 conn(0x7f20e40b7ea0 msgr2=0x7f20e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:42.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 1 -- 172.21.15.19:0/22619683 shutdown_connections 2024-03-25T19:29:42.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 1 -- 172.21.15.19:0/22619683 wait complete. 2024-03-25T19:29:42.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:42.142+0000 7f20eb94b640 1 librados: shutdown 2024-03-25T19:29:42.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:48.960 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.071+0000 7fb8b0b90640 10 monclient: get_monmap_and_config 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.071+0000 7fb8b0b90640 10 monclient: build_initial_monmap 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.071+0000 7fb8b0b90640 10 monclient: monmap: 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:29:49.072369+0000 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:29:49.072369+0000 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:29:49.071 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:49.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.071+0000 7fb8b0b90640 1 Processor -- start 2024-03-25T19:29:49.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.072+0000 7fb8b0b90640 1 -- start start 2024-03-25T19:29:49.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.072+0000 7fb8b0b90640 10 monclient: init 2024-03-25T19:29:49.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.072+0000 7fb8b0b90640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:49.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.072+0000 7fb8b0b90640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:49.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.072+0000 7fb8b0b90640 10 monclient(hunting): picked mon.noname-a con 0x7fb8ac0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:49.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.072+0000 7fb8b0b90640 10 monclient(hunting): picked mon.noname-b con 0x7fb8ac13f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:49.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.072+0000 7fb8b0b90640 10 monclient(hunting): picked mon.noname-c con 0x7fb8ac1430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:49.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.073+0000 7fb8b0b90640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac142520 con 0x7fb8ac0a98f0 2024-03-25T19:29:49.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.073+0000 7fb8b0b90640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac0b28a0 con 0x7fb8ac1430c0 2024-03-25T19:29:49.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.073+0000 7fb8b0b90640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac0b2a30 con 0x7fb8ac13f710 2024-03-25T19:29:49.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.073+0000 7fb8b0b90640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:49.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.073+0000 7fb8b0b90640 10 monclient(hunting): authenticate will time out at 6042.343013s 2024-03-25T19:29:49.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.078+0000 7fb8aad9d640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fb8ac1430c0 0x7fb8ac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:50556/0 (socket says 172.21.15.19:50556) 2024-03-25T19:29:49.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.078+0000 7fb8aad9d640 1 -- 172.21.15.19:0/3850255064 learned_addr learned my addr 172.21.15.19:0/3850255064 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:49.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.099+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571473195 0 0) 0x7fb8ac142520 con 0x7fb8ac0a98f0 2024-03-25T19:29:49.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.099+0000 7fb8a959a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:49.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.099+0000 7fb8a959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:49.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.099+0000 7fb8a959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:49.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.099+0000 7fb8a959a640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:29:49.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.099+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb894003520 con 0x7fb8ac0a98f0 2024-03-25T19:29:49.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.103+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395671975 0 0) 0x7fb8ac0b28a0 con 0x7fb8ac1430c0 2024-03-25T19:29:49.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.103+0000 7fb8a959a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:49.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.103+0000 7fb8a959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:49.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.103+0000 7fb8a959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:49.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.103+0000 7fb8a959a640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:29:49.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.103+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8ac142520 con 0x7fb8ac1430c0 2024-03-25T19:29:49.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.115+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685415269 0 0) 0x7fb8ac0b2a30 con 0x7fb8ac13f710 2024-03-25T19:29:49.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.115+0000 7fb8a959a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:49.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.115+0000 7fb8a959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:49.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.115+0000 7fb8a959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:49.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.115+0000 7fb8a959a640 10 monclient(hunting): my global_id is 4244 2024-03-25T19:29:49.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.115+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8ac0b28a0 con 0x7fb8ac13f710 2024-03-25T19:29:49.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3224700346 0 0) 0x7fb894003520 con 0x7fb8ac0a98f0 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 >> v1:172.21.15.19:6790/0 conn(0x7fb8ac13f710 legacy=0x7fb8ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 >> v1:172.21.15.188:6789/0 conn(0x7fb8ac1430c0 legacy=0x7fb8ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 10 monclient: _finish_hunting 0 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 1 monclient: found mon.noname-a 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0b44c0 con 0x7fb8ac0a98f0 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 10 monclient: _finish_auth 0 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 10 monclient: _check_auth_tickets 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb89c001690 con 0x7fb8ac0a98f0 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:49.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8a959a640 10 monclient: dump: 2024-03-25T19:29:49.123 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:49.123 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:49.123 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:49.123 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:49.123 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:49.123 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:49.123 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:49.123 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:49.123 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:49.123 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:49.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8b0b90640 5 monclient: authenticate success, global_id 4260 2024-03-25T19:29:49.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.121+0000 7fb8b0b90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:49.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.137+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb89c0017f0 con 0x7fb8ac0a98f0 2024-03-25T19:29:49.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.137+0000 7fb8a959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:49.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.137+0000 7fb8a959a640 1 -- 172.21.15.19:0/3850255064 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb89c001c00 con 0x7fb8ac0a98f0 2024-03-25T19:29:49.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.137+0000 7fb8b0b90640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:49.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.137+0000 7fb8b0b90640 4 set_mon_vals no callback set 2024-03-25T19:29:49.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.137+0000 7fb8a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:49.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.137+0000 7fb8a959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:49.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.137+0000 7fb8a959a640 10 monclient: dump: 2024-03-25T19:29:49.138 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:49.138 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:49.138 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:49.138 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:49.138 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:49.138 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:49.138 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:49.138 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:49.138 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:49.138 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:49.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.138+0000 7fb8b0b90640 10 monclient: shutdown 2024-03-25T19:29:49.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.138+0000 7fb8b0b90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:49.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.138+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3850255064 >> v1:172.21.15.19:6789/0 conn(0x7fb8ac0a98f0 legacy=0x7fb8ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:49.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.138+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3850255064 shutdown_connections 2024-03-25T19:29:49.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.138+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3850255064 >> 172.21.15.19:0/3850255064 conn(0x7fb8ac0b7ea0 msgr2=0x7fb8ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:49.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.138+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3850255064 shutdown_connections 2024-03-25T19:29:49.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.138+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3850255064 wait complete. 2024-03-25T19:29:49.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.139+0000 7fb8b0b90640 10 monclient: build_initial_monmap 2024-03-25T19:29:49.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.139+0000 7fb8b0b90640 10 monclient: monmap: 2024-03-25T19:29:49.140 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:49.140 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:49.140 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.139+0000 7fb8b0b90640 1 librados: starting msgr at 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.139+0000 7fb8b0b90640 1 librados: starting objecter 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.140+0000 7fb8b0b90640 1 Processor -- start 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.140+0000 7fb8b0b90640 1 -- start start 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.140+0000 7fb8b0b90640 1 librados: setting wanted keys 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.140+0000 7fb8b0b90640 1 librados: calling monclient init 2024-03-25T19:29:49.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.140+0000 7fb8b0b90640 10 monclient: init 2024-03-25T19:29:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.140+0000 7fb8b0b90640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.140+0000 7fb8b0b90640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.140+0000 7fb8b0b90640 10 monclient(hunting): picked mon.noname-a con 0x7fb8ac0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.140+0000 7fb8b0b90640 10 monclient(hunting): picked mon.noname-c con 0x7fb8ac13f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.140+0000 7fb8b0b90640 10 monclient(hunting): picked mon.noname-b con 0x7fb8ac1430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.140+0000 7fb8b0b90640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac1db050 con 0x7fb8ac0a98f0 2024-03-25T19:29:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.141+0000 7fb8b0b90640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac1dab70 con 0x7fb8ac1430c0 2024-03-25T19:29:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.141+0000 7fb8b0b90640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac1dad00 con 0x7fb8ac13f710 2024-03-25T19:29:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.141+0000 7fb8b0b90640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.141+0000 7fb8b0b90640 10 monclient(hunting): authenticate will time out at 6042.410982s 2024-03-25T19:29:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.143+0000 7fb8aad9d640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fb8ac1430c0 0x7fb8ac1393b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:50596/0 (socket says 172.21.15.19:50596) 2024-03-25T19:29:49.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.143+0000 7fb8aad9d640 1 -- 172.21.15.19:0/3555304425 learned_addr learned my addr 172.21.15.19:0/3555304425 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:49.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.143+0000 7fb89affd640 10 client.?.objecter ms_handle_connect 0x7fb8ac1430c0 2024-03-25T19:29:49.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.143+0000 7fb89affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.151+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339322922 0 0) 0x7fb8ac1dab70 con 0x7fb8ac1430c0 2024-03-25T19:29:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.151+0000 7fb89affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.151+0000 7fb89affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.151+0000 7fb89affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.151+0000 7fb89affd640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:29:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.151+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb88c004eb0 con 0x7fb8ac1430c0 2024-03-25T19:29:49.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.157+0000 7fb89affd640 10 client.?.objecter ms_handle_connect 0x7fb8ac13f710 2024-03-25T19:29:49.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.157+0000 7fb89affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:49.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.168+0000 7fb89affd640 10 client.?.objecter ms_handle_connect 0x7fb8ac0a98f0 2024-03-25T19:29:49.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.168+0000 7fb89affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:49.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.169+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3406360545 0 0) 0x7fb88c004eb0 con 0x7fb8ac1430c0 2024-03-25T19:29:49.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.169+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8ac1dab70 con 0x7fb8ac1430c0 2024-03-25T19:29:49.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.169+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb8a4001c70 con 0x7fb8ac1430c0 2024-03-25T19:29:49.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.169+0000 7fb89affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:49.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.169+0000 7fb89affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:49.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.169+0000 7fb89affd640 10 monclient(hunting): dump: 2024-03-25T19:29:49.169 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:49.169 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:49.169 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:49.170 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:49.170 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:49.170 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:49.170 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:49.170 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:49.170 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:49.170 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:49.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.169+0000 7fb89affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:49.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.172+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276515625 0 0) 0x7fb8ac1dad00 con 0x7fb8ac13f710 2024-03-25T19:29:49.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.172+0000 7fb89affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:49.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.172+0000 7fb89affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:49.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.172+0000 7fb89affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:49.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.172+0000 7fb89affd640 10 monclient(hunting): my global_id is 4247 2024-03-25T19:29:49.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.172+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb88c004eb0 con 0x7fb8ac13f710 2024-03-25T19:29:49.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.189+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3761286432 0 0) 0x7fb88c004eb0 con 0x7fb8ac13f710 2024-03-25T19:29:49.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.189+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8ac1dad00 con 0x7fb8ac13f710 2024-03-25T19:29:49.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.189+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb8a0001830 con 0x7fb8ac13f710 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.189+0000 7fb89affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.189+0000 7fb89affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.189+0000 7fb89affd640 10 monclient(hunting): dump: 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:49.190 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.191+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277677957 0 0) 0x7fb8ac1db050 con 0x7fb8ac0a98f0 2024-03-25T19:29:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.191+0000 7fb89affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.191+0000 7fb89affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.191+0000 7fb89affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.191+0000 7fb89affd640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:29:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.191+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb88c004eb0 con 0x7fb8ac0a98f0 2024-03-25T19:29:49.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2345132017 0 0) 0x7fb8ac1dab70 con 0x7fb8ac1430c0 2024-03-25T19:29:49.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 >> v1:172.21.15.19:6790/0 conn(0x7fb8ac13f710 legacy=0x7fb8ac1d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:49.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 >> v1:172.21.15.19:6789/0 conn(0x7fb8ac0a98f0 legacy=0x7fb8ac1db4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:49.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb89affd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:49.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb89affd640 1 monclient: found mon.b 2024-03-25T19:29:49.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb89affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb89affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0b70a0 con 0x7fb8ac1430c0 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb89affd640 10 monclient: _finish_auth 0 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb89affd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb89affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb8b0b90640 5 monclient: authenticate success, global_id 4258 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb8b0b90640 10 monclient: _renew_subs 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb8b0b90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3555304425 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ac0b72f0 con 0x7fb8ac1430c0 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb8b0b90640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb8b0b90640 10 monclient: _renew_subs 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb8b0b90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3555304425 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ac1e32a0 con 0x7fb8ac1430c0 2024-03-25T19:29:49.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.196+0000 7fb8b0b90640 1 librados: init done 2024-03-25T19:29:49.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.197+0000 7fb8b0b90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:49.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.197+0000 7fb8b0b90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:49.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.197+0000 7fb8b0b90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:49.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.197+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3555304425 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ac0627b0 con 0x7fb8ac1430c0 2024-03-25T19:29:49.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.481+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8a4004b60 con 0x7fb8ac1430c0 2024-03-25T19:29:49.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.481+0000 7fb89affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:49.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.481+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb8a4004e70 con 0x7fb8ac1430c0 2024-03-25T19:29:49.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.481+0000 7fb89affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:49.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.481+0000 7fb89affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:49.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.481+0000 7fb89affd640 10 monclient: dump: 2024-03-25T19:29:49.481 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:49.481 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:49.481 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:49.482 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:49.482 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:49.482 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:49.482 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:49.482 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:49.482 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:49.482 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:49.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.481+0000 7fb8a8d99640 4 set_mon_vals no callback set 2024-03-25T19:29:49.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.482+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fb8a401ae40 con 0x7fb8ac1430c0 2024-03-25T19:29:49.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.482+0000 7fb8aa59c640 10 monclient: get_auth_request con 0x7fb88c06dfb0 auth_method 0 2024-03-25T19:29:49.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.483+0000 7fb89affd640 10 client.4258.objecter ms_handle_connect 0x7fb88c06dfb0 2024-03-25T19:29:49.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.487+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fb8a4054b40 con 0x7fb8ac1430c0 2024-03-25T19:29:49.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.487+0000 7fb89affd640 10 client.4258.objecter ms_dispatch 0x7fb8ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:49.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.487+0000 7fb89affd640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:49.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.487+0000 7fb89affd640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:49.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.487+0000 7fb89affd640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-25T19:29:49.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.488+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8a4050480 con 0x7fb8ac1430c0 2024-03-25T19:29:49.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.488+0000 7fb89affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:49.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.488+0000 7fb89affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:49.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.746+0000 7fb8b0b90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:49.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.746+0000 7fb8b0b90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:49.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.746+0000 7fb8b0b90640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:29:49.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.746+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3555304425 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8ac1db200 con 0x7fb8ac1430c0 2024-03-25T19:29:49.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.784+0000 7fb89affd640 1 -- 172.21.15.19:0/3555304425 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fb8ac1db200 con 0x7fb8ac1430c0 2024-03-25T19:29:49.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.784+0000 7fb89affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:49.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.784+0000 7fb89affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:49.785 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:29:49.785 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:49.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 10 librados: watch_flush enter 2024-03-25T19:29:49.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 10 librados: watch_flush exit 2024-03-25T19:29:49.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-25T19:29:49.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 10 client.4258.objecter successfully canceled tick 2024-03-25T19:29:49.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3555304425 >> v1:172.21.15.188:6800/2207506229 conn(0x7fb88c06dfb0 legacy=0x7fb88c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:49.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 10 monclient: shutdown 2024-03-25T19:29:49.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:49.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3555304425 >> v1:172.21.15.188:6789/0 conn(0x7fb8ac1430c0 legacy=0x7fb8ac1393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:49.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3555304425 shutdown_connections 2024-03-25T19:29:49.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3555304425 >> 172.21.15.19:0/3555304425 conn(0x7fb8ac0b7ea0 msgr2=0x7fb8ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:49.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3555304425 shutdown_connections 2024-03-25T19:29:49.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 1 -- 172.21.15.19:0/3555304425 wait complete. 2024-03-25T19:29:49.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:49.786+0000 7fb8b0b90640 1 librados: shutdown 2024-03-25T19:29:49.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:51.802 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:14.531 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:51.827 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:01:14.530 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:51.860 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:14.590 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:51.880 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:01:14.583 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:51.923 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:14.653 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:51.995 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:14.724 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:52.100 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:14.830 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:52.112 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:01:14.815 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:52.239 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:14.968 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:56.605 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:56.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.722+0000 7fd2c8e0e640 10 monclient: get_monmap_and_config 2024-03-25T19:29:56.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.722+0000 7fd2c8e0e640 10 monclient: build_initial_monmap 2024-03-25T19:29:56.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.722+0000 7fd2c8e0e640 10 monclient: monmap: 2024-03-25T19:29:56.723 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:56.723 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:56.723 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:29:56.723936+0000 2024-03-25T19:29:56.723 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:29:56.723936+0000 2024-03-25T19:29:56.723 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:56.723 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:56.723 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:56.723 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:29:56.723 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:29:56.723 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:56.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.723+0000 7fd2c8e0e640 1 Processor -- start 2024-03-25T19:29:56.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.723+0000 7fd2c8e0e640 1 -- start start 2024-03-25T19:29:56.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.723+0000 7fd2c8e0e640 10 monclient: init 2024-03-25T19:29:56.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.724+0000 7fd2c8e0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:56.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.724+0000 7fd2c8e0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:56.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.724+0000 7fd2c8e0e640 10 monclient(hunting): picked mon.noname-c con 0x7fd2c40a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:56.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.724+0000 7fd2c8e0e640 10 monclient(hunting): picked mon.noname-b con 0x7fd2c413f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:56.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.724+0000 7fd2c8e0e640 10 monclient(hunting): picked mon.noname-a con 0x7fd2c41430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:56.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.724+0000 7fd2c8e0e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4142520 con 0x7fd2c41430c0 2024-03-25T19:29:56.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.724+0000 7fd2c8e0e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40b28a0 con 0x7fd2c40a98f0 2024-03-25T19:29:56.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.724+0000 7fd2c8e0e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40b2a30 con 0x7fd2c413f710 2024-03-25T19:29:56.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.724+0000 7fd2c8e0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:56.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.724+0000 7fd2c8e0e640 10 monclient(hunting): authenticate will time out at 6049.994808s 2024-03-25T19:29:56.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.742+0000 7fd2c259c640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fd2c40a98f0 0x7fd2c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:59696/0 (socket says 172.21.15.19:59696) 2024-03-25T19:29:56.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.742+0000 7fd2c259c640 1 -- 172.21.15.19:0/2812024060 learned_addr learned my addr 172.21.15.19:0/2812024060 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:56.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.757+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619195455 0 0) 0x7fd2c40b28a0 con 0x7fd2c40a98f0 2024-03-25T19:29:56.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.757+0000 7fd2c159a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.757+0000 7fd2c159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.757+0000 7fd2c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.757+0000 7fd2c159a640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:29:56.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.757+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ac003520 con 0x7fd2c40a98f0 2024-03-25T19:29:56.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.760+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803636439 0 0) 0x7fd2c40b2a30 con 0x7fd2c413f710 2024-03-25T19:29:56.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.760+0000 7fd2c159a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.760+0000 7fd2c159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.760+0000 7fd2c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.760+0000 7fd2c159a640 10 monclient(hunting): my global_id is 4250 2024-03-25T19:29:56.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.761+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c40b28a0 con 0x7fd2c413f710 2024-03-25T19:29:56.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.762+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427421748 0 0) 0x7fd2c4142520 con 0x7fd2c41430c0 2024-03-25T19:29:56.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.762+0000 7fd2c159a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.762+0000 7fd2c159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.762+0000 7fd2c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.762+0000 7fd2c159a640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:29:56.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.762+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c40b2a30 con 0x7fd2c41430c0 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368887538 0 0) 0x7fd2ac003520 con 0x7fd2c40a98f0 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 >> v1:172.21.15.19:6790/0 conn(0x7fd2c413f710 legacy=0x7fd2c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 >> v1:172.21.15.19:6789/0 conn(0x7fd2c41430c0 legacy=0x7fd2c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 10 monclient: _finish_hunting 0 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 1 monclient: found mon.noname-c 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c40b44c0 con 0x7fd2c40a98f0 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 10 monclient: _finish_auth 0 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 10 monclient: _check_auth_tickets 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd2b4001690 con 0x7fd2c40a98f0 2024-03-25T19:29:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c8e0e640 5 monclient: authenticate success, global_id 4261 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c8e0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c159a640 10 monclient: dump: 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:56.772 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.771+0000 7fd2c8e0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.786+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2b40017f0 con 0x7fd2c40a98f0 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.786+0000 7fd2c159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.786+0000 7fd2c159a640 1 -- 172.21.15.19:0/2812024060 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd2b4001c00 con 0x7fd2c40a98f0 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.786+0000 7fd2c8e0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.786+0000 7fd2c8e0e640 4 set_mon_vals no callback set 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.786+0000 7fd2c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.786+0000 7fd2c159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.786+0000 7fd2c159a640 10 monclient: dump: 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:56.786 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.787 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:56.787 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:56.787 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:56.787 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:56.787 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:56.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.786+0000 7fd2c8e0e640 10 monclient: shutdown 2024-03-25T19:29:56.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.786+0000 7fd2c8e0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:56.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.786+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/2812024060 >> v1:172.21.15.188:6789/0 conn(0x7fd2c40a98f0 legacy=0x7fd2c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.787+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/2812024060 shutdown_connections 2024-03-25T19:29:56.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.787+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/2812024060 >> 172.21.15.19:0/2812024060 conn(0x7fd2c40b7ea0 msgr2=0x7fd2c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:56.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.787+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/2812024060 shutdown_connections 2024-03-25T19:29:56.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.787+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/2812024060 wait complete. 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.788+0000 7fd2c8e0e640 10 monclient: build_initial_monmap 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.788+0000 7fd2c8e0e640 10 monclient: monmap: 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.788+0000 7fd2c8e0e640 1 librados: starting msgr at 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.788+0000 7fd2c8e0e640 1 librados: starting objecter 2024-03-25T19:29:56.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.788+0000 7fd2c8e0e640 1 Processor -- start 2024-03-25T19:29:56.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.788+0000 7fd2c8e0e640 1 -- start start 2024-03-25T19:29:56.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.788+0000 7fd2c8e0e640 1 librados: setting wanted keys 2024-03-25T19:29:56.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.788+0000 7fd2c8e0e640 1 librados: calling monclient init 2024-03-25T19:29:56.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.788+0000 7fd2c8e0e640 10 monclient: init 2024-03-25T19:29:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.789+0000 7fd2c8e0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.789+0000 7fd2c8e0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.789+0000 7fd2c8e0e640 10 monclient(hunting): picked mon.noname-a con 0x7fd2c40a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:29:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.789+0000 7fd2c8e0e640 10 monclient(hunting): picked mon.noname-c con 0x7fd2c413f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:29:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.789+0000 7fd2c8e0e640 10 monclient(hunting): picked mon.noname-b con 0x7fd2c41430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:29:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.789+0000 7fd2c8e0e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4138250 con 0x7fd2c40a98f0 2024-03-25T19:29:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.789+0000 7fd2c8e0e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4137d50 con 0x7fd2c41430c0 2024-03-25T19:29:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.789+0000 7fd2c8e0e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4137ee0 con 0x7fd2c413f710 2024-03-25T19:29:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.789+0000 7fd2c8e0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.789+0000 7fd2c8e0e640 10 monclient(hunting): authenticate will time out at 6050.059225s 2024-03-25T19:29:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.802+0000 7fd2c1d9b640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fd2c413f710 0x7fd2c41d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:46456/0 (socket says 172.21.15.19:46456) 2024-03-25T19:29:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.802+0000 7fd2c1d9b640 1 -- 172.21.15.19:0/1969709611 learned_addr learned my addr 172.21.15.19:0/1969709611 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:29:56.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.809+0000 7fd2b2ffd640 10 client.?.objecter ms_handle_connect 0x7fd2c413f710 2024-03-25T19:29:56.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.809+0000 7fd2b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:56.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.812+0000 7fd2b2ffd640 10 client.?.objecter ms_handle_connect 0x7fd2c40a98f0 2024-03-25T19:29:56.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.812+0000 7fd2b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:56.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.813+0000 7fd2b2ffd640 10 client.?.objecter ms_handle_connect 0x7fd2c41430c0 2024-03-25T19:29:56.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.813+0000 7fd2b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:56.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.816+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889412125 0 0) 0x7fd2c4138250 con 0x7fd2c40a98f0 2024-03-25T19:29:56.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.816+0000 7fd2b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.816+0000 7fd2b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.816+0000 7fd2b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.816+0000 7fd2b2ffd640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:29:56.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.816+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2a4004eb0 con 0x7fd2c40a98f0 2024-03-25T19:29:56.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.825+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588592405 0 0) 0x7fd2c4137d50 con 0x7fd2c41430c0 2024-03-25T19:29:56.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.825+0000 7fd2b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.825+0000 7fd2b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.825+0000 7fd2b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.825+0000 7fd2b2ffd640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:29:56.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.825+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c4138250 con 0x7fd2c41430c0 2024-03-25T19:29:56.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.827+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 442478341 0 0) 0x7fd2a4004eb0 con 0x7fd2c40a98f0 2024-03-25T19:29:56.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.827+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2c4137d50 con 0x7fd2c40a98f0 2024-03-25T19:29:56.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.827+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd2b4001b00 con 0x7fd2c40a98f0 2024-03-25T19:29:56.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.827+0000 7fd2b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.827+0000 7fd2b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.827+0000 7fd2b2ffd640 10 monclient(hunting): dump: 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:56.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.827+0000 7fd2b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:56.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.829+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088602250 0 0) 0x7fd2c4137ee0 con 0x7fd2c413f710 2024-03-25T19:29:56.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.829+0000 7fd2b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.829+0000 7fd2b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.829+0000 7fd2b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.829+0000 7fd2b2ffd640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:29:56.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.829+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2a4004eb0 con 0x7fd2c413f710 2024-03-25T19:29:56.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.839+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1218660784 0 0) 0x7fd2c4138250 con 0x7fd2c41430c0 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.839+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2c4137ee0 con 0x7fd2c41430c0 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.839+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd2bc001b50 con 0x7fd2c41430c0 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.839+0000 7fd2b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.839+0000 7fd2b2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.839+0000 7fd2b2ffd640 10 monclient(hunting): dump: 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:56.839 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:56.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 550640296 0 0) 0x7fd2c4137d50 con 0x7fd2c40a98f0 2024-03-25T19:29:56.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 >> v1:172.21.15.19:6790/0 conn(0x7fd2c413f710 legacy=0x7fd2c41d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 >> v1:172.21.15.188:6789/0 conn(0x7fd2c41430c0 legacy=0x7fd2c41d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:56.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2b2ffd640 1 monclient: found mon.a 2024-03-25T19:29:56.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:56.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2b2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:56.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c40b7730 con 0x7fd2c40a98f0 2024-03-25T19:29:56.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2b2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:56.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:56.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:56.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2c8e0e640 5 monclient: authenticate success, global_id 4269 2024-03-25T19:29:56.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2c8e0e640 10 monclient: _renew_subs 2024-03-25T19:29:56.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2c8e0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:56.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/1969709611 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c40b7920 con 0x7fd2c40a98f0 2024-03-25T19:29:56.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2c8e0e640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:56.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2c8e0e640 10 monclient: _renew_subs 2024-03-25T19:29:56.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2c8e0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:56.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/1969709611 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c41e33d0 con 0x7fd2c40a98f0 2024-03-25T19:29:56.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.842+0000 7fd2c8e0e640 1 librados: init done 2024-03-25T19:29:56.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.843+0000 7fd2c8e0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:56.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.843+0000 7fd2c8e0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:56.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.843+0000 7fd2c8e0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:56.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:56.843+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/1969709611 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c40627b0 con 0x7fd2c40a98f0 2024-03-25T19:29:57.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.167+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2b4001220 con 0x7fd2c40a98f0 2024-03-25T19:29:57.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.167+0000 7fd2b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:57.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.167+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd2b4004c30 con 0x7fd2c40a98f0 2024-03-25T19:29:57.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.167+0000 7fd2b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:57.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.167+0000 7fd2b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:57.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.167+0000 7fd2b2ffd640 10 monclient: dump: 2024-03-25T19:29:57.168 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:29:57.168 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:29:57.168 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:57.168 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:29:57.168 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:57.168 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:29:57.168 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:29:57.168 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:29:57.168 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:29:57.168 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:29:57.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.167+0000 7fd2c0d99640 4 set_mon_vals no callback set 2024-03-25T19:29:57.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.168+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fd2b401ac00 con 0x7fd2c40a98f0 2024-03-25T19:29:57.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.169+0000 7fd2c1d9b640 10 monclient: get_auth_request con 0x7fd2a406dfb0 auth_method 0 2024-03-25T19:29:57.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.169+0000 7fd2b2ffd640 10 client.4269.objecter ms_handle_connect 0x7fd2a406dfb0 2024-03-25T19:29:57.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.176+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fd2b40548a0 con 0x7fd2c40a98f0 2024-03-25T19:29:57.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.176+0000 7fd2b2ffd640 10 client.4269.objecter ms_dispatch 0x7fd2c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:57.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.176+0000 7fd2b2ffd640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:57.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.176+0000 7fd2b2ffd640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:57.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.176+0000 7fd2b2ffd640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-25T19:29:57.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.176+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2b40501e0 con 0x7fd2c40a98f0 2024-03-25T19:29:57.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.176+0000 7fd2b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:57.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.176+0000 7fd2b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:57.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.401+0000 7fd2c8e0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:57.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.401+0000 7fd2c8e0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:57.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.401+0000 7fd2c8e0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:29:57.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.402+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/1969709611 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2c413df50 con 0x7fd2c40a98f0 2024-03-25T19:29:57.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.423+0000 7fd2b2ffd640 1 -- 172.21.15.19:0/1969709611 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fd2b40522e0 con 0x7fd2c40a98f0 2024-03-25T19:29:57.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.423+0000 7fd2b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:57.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.423+0000 7fd2b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:57.424 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:29:57.424 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:57.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 10 librados: watch_flush enter 2024-03-25T19:29:57.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 10 librados: watch_flush exit 2024-03-25T19:29:57.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-25T19:29:57.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 10 client.4269.objecter successfully canceled tick 2024-03-25T19:29:57.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/1969709611 >> v1:172.21.15.188:6800/2207506229 conn(0x7fd2a406dfb0 legacy=0x7fd2a4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:57.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 10 monclient: shutdown 2024-03-25T19:29:57.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:57.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/1969709611 >> v1:172.21.15.19:6789/0 conn(0x7fd2c40a98f0 legacy=0x7fd2c4137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:57.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/1969709611 shutdown_connections 2024-03-25T19:29:57.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/1969709611 >> 172.21.15.19:0/1969709611 conn(0x7fd2c40b7ea0 msgr2=0x7fd2c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:57.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/1969709611 shutdown_connections 2024-03-25T19:29:57.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 1 -- 172.21.15.19:0/1969709611 wait complete. 2024-03-25T19:29:57.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:29:57.426+0000 7fd2c8e0e640 1 librados: shutdown 2024-03-25T19:29:57.436 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:04.242 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:04.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.358+0000 7efe5dfeb640 10 monclient: get_monmap_and_config 2024-03-25T19:30:04.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.358+0000 7efe5dfeb640 10 monclient: build_initial_monmap 2024-03-25T19:30:04.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.358+0000 7efe5dfeb640 10 monclient: monmap: 2024-03-25T19:30:04.358 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:04.358 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:04.358 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:30:04.359643+0000 2024-03-25T19:30:04.358 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:30:04.359643+0000 2024-03-25T19:30:04.358 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:04.359 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:04.359 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:04.359 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:30:04.359 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:30:04.359 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:04.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.359+0000 7efe5dfeb640 1 Processor -- start 2024-03-25T19:30:04.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.359+0000 7efe5dfeb640 1 -- start start 2024-03-25T19:30:04.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.359+0000 7efe5dfeb640 10 monclient: init 2024-03-25T19:30:04.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.359+0000 7efe5dfeb640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:04.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.359+0000 7efe5dfeb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:04.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.359+0000 7efe5dfeb640 10 monclient(hunting): picked mon.noname-a con 0x7efe580a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:04.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.359+0000 7efe5dfeb640 10 monclient(hunting): picked mon.noname-b con 0x7efe5813f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:04.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.359+0000 7efe5dfeb640 10 monclient(hunting): picked mon.noname-c con 0x7efe581430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:04.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.359+0000 7efe5dfeb640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe58142520 con 0x7efe580a98f0 2024-03-25T19:30:04.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.360+0000 7efe5dfeb640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe580b28a0 con 0x7efe581430c0 2024-03-25T19:30:04.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.360+0000 7efe5dfeb640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe580b2a30 con 0x7efe5813f710 2024-03-25T19:30:04.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.360+0000 7efe5dfeb640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:04.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.360+0000 7efe5dfeb640 10 monclient(hunting): authenticate will time out at 6057.629956s 2024-03-25T19:30:04.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.371+0000 7efe56ffd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7efe5813f710 0x7efe581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:58936/0 (socket says 172.21.15.19:58936) 2024-03-25T19:30:04.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.371+0000 7efe56ffd640 1 -- 172.21.15.19:0/2183901869 learned_addr learned my addr 172.21.15.19:0/2183901869 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:04.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.385+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517683612 0 0) 0x7efe580b28a0 con 0x7efe581430c0 2024-03-25T19:30:04.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.385+0000 7efe567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.385+0000 7efe567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.385+0000 7efe567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.385+0000 7efe567fc640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:30:04.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.385+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe40003520 con 0x7efe581430c0 2024-03-25T19:30:04.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.386+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732449294 0 0) 0x7efe58142520 con 0x7efe580a98f0 2024-03-25T19:30:04.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.386+0000 7efe567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.386+0000 7efe567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.386+0000 7efe567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.386+0000 7efe567fc640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:30:04.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.386+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe580b28a0 con 0x7efe580a98f0 2024-03-25T19:30:04.393 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.393+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811366624 0 0) 0x7efe580b2a30 con 0x7efe5813f710 2024-03-25T19:30:04.393 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.393+0000 7efe567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.393 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.393+0000 7efe567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.394 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.393+0000 7efe567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.394 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.393+0000 7efe567fc640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:30:04.394 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.393+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe58142520 con 0x7efe5813f710 2024-03-25T19:30:04.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1345755352 0 0) 0x7efe40003520 con 0x7efe581430c0 2024-03-25T19:30:04.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 >> v1:172.21.15.19:6790/0 conn(0x7efe5813f710 legacy=0x7efe581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 >> v1:172.21.15.19:6789/0 conn(0x7efe580a98f0 legacy=0x7efe580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 1 monclient: found mon.noname-c 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe580b44c0 con 0x7efe581430c0 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efe50001a10 con 0x7efe581430c0 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe567fc640 10 monclient: dump: 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:04.401 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:04.401 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:04.401 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:04.401 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:04.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe5dfeb640 5 monclient: authenticate success, global_id 4267 2024-03-25T19:30:04.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.399+0000 7efe5dfeb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.414+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe50001b70 con 0x7efe581430c0 2024-03-25T19:30:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.414+0000 7efe567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.414+0000 7efe567fc640 1 -- 172.21.15.19:0/2183901869 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efe50001f80 con 0x7efe581430c0 2024-03-25T19:30:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.414+0000 7efe567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.414+0000 7efe567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.414+0000 7efe567fc640 10 monclient: dump: 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.414+0000 7efe5dfeb640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.414+0000 7efe5dfeb640 4 set_mon_vals no callback set 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.415+0000 7efe5dfeb640 10 monclient: shutdown 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.415+0000 7efe5dfeb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.415+0000 7efe5dfeb640 1 -- 172.21.15.19:0/2183901869 >> v1:172.21.15.188:6789/0 conn(0x7efe581430c0 legacy=0x7efe5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.415+0000 7efe5dfeb640 1 -- 172.21.15.19:0/2183901869 shutdown_connections 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.415+0000 7efe5dfeb640 1 -- 172.21.15.19:0/2183901869 >> 172.21.15.19:0/2183901869 conn(0x7efe580b7ea0 msgr2=0x7efe580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.415+0000 7efe5dfeb640 1 -- 172.21.15.19:0/2183901869 shutdown_connections 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.415+0000 7efe5dfeb640 1 -- 172.21.15.19:0/2183901869 wait complete. 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.416+0000 7efe5dfeb640 10 monclient: build_initial_monmap 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.416+0000 7efe5dfeb640 10 monclient: monmap: 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:30:04.418 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.416+0000 7efe5dfeb640 1 librados: starting msgr at 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.416+0000 7efe5dfeb640 1 librados: starting objecter 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.416+0000 7efe5dfeb640 1 Processor -- start 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 1 -- start start 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 1 librados: setting wanted keys 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 1 librados: calling monclient init 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 10 monclient: init 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:04.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:04.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 10 monclient(hunting): picked mon.noname-b con 0x7efe580a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:04.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 10 monclient(hunting): picked mon.noname-c con 0x7efe5813f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:04.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 10 monclient(hunting): picked mon.noname-a con 0x7efe581430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:04.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe581369d0 con 0x7efe581430c0 2024-03-25T19:30:04.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe58133a60 con 0x7efe580a98f0 2024-03-25T19:30:04.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe580ab9e0 con 0x7efe5813f710 2024-03-25T19:30:04.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:04.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.417+0000 7efe5dfeb640 10 monclient(hunting): authenticate will time out at 6057.687884s 2024-03-25T19:30:04.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.429+0000 7efe56ffd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7efe5813f710 0x7efe581362c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:59002/0 (socket says 172.21.15.19:59002) 2024-03-25T19:30:04.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.429+0000 7efe56ffd640 1 -- 172.21.15.19:0/1791964115 learned_addr learned my addr 172.21.15.19:0/1791964115 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:04.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.437+0000 7efe3ffff640 10 client.?.objecter ms_handle_connect 0x7efe5813f710 2024-03-25T19:30:04.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.437+0000 7efe3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:04.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.439+0000 7efe3ffff640 10 client.?.objecter ms_handle_connect 0x7efe580a98f0 2024-03-25T19:30:04.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.439+0000 7efe3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:04.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.449+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659374181 0 0) 0x7efe58133a60 con 0x7efe580a98f0 2024-03-25T19:30:04.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.449+0000 7efe3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.449+0000 7efe3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.449+0000 7efe3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.449+0000 7efe3ffff640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:30:04.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.449+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe44004eb0 con 0x7efe580a98f0 2024-03-25T19:30:04.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.450+0000 7efe3ffff640 10 client.?.objecter ms_handle_connect 0x7efe581430c0 2024-03-25T19:30:04.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.450+0000 7efe3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:04.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.456+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694129674 0 0) 0x7efe580ab9e0 con 0x7efe5813f710 2024-03-25T19:30:04.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.456+0000 7efe3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.456+0000 7efe3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.456+0000 7efe3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.456+0000 7efe3ffff640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:30:04.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.456+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe58133a60 con 0x7efe5813f710 2024-03-25T19:30:04.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.463+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2587464395 0 0) 0x7efe44004eb0 con 0x7efe580a98f0 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.463+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe580ab9e0 con 0x7efe580a98f0 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.463+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efe48001970 con 0x7efe580a98f0 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.463+0000 7efe3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.463+0000 7efe3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.463+0000 7efe3ffff640 10 monclient(hunting): dump: 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:04.464 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:04.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.463+0000 7efe3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:04.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.473+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092039429 0 0) 0x7efe581369d0 con 0x7efe581430c0 2024-03-25T19:30:04.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.473+0000 7efe3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.473+0000 7efe3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.473+0000 7efe3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.473+0000 7efe3ffff640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:30:04.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.473+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe44004eb0 con 0x7efe581430c0 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.477+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 144583731 0 0) 0x7efe58133a60 con 0x7efe5813f710 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.477+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe581369d0 con 0x7efe5813f710 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.477+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efe4c0018f0 con 0x7efe5813f710 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.477+0000 7efe3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.477+0000 7efe3ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.477+0000 7efe3ffff640 10 monclient(hunting): dump: 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:04.477 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:04.478 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:04.478 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1513031087 0 0) 0x7efe580ab9e0 con 0x7efe580a98f0 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 >> v1:172.21.15.19:6790/0 conn(0x7efe5813f710 legacy=0x7efe581362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 >> v1:172.21.15.19:6789/0 conn(0x7efe581430c0 legacy=0x7efe58133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe3ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe3ffff640 1 monclient: found mon.b 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe3ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe580abcc0 con 0x7efe580a98f0 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe3ffff640 10 monclient: _finish_auth 0 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe3ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe5dfeb640 5 monclient: authenticate success, global_id 4270 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe5dfeb640 10 monclient: _renew_subs 2024-03-25T19:30:04.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe5dfeb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:04.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe5dfeb640 1 -- 172.21.15.19:0/1791964115 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe580abfa0 con 0x7efe580a98f0 2024-03-25T19:30:04.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe5dfeb640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:04.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe5dfeb640 10 monclient: _renew_subs 2024-03-25T19:30:04.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe5dfeb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:04.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe5dfeb640 1 -- 172.21.15.19:0/1791964115 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe58137e40 con 0x7efe580a98f0 2024-03-25T19:30:04.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.482+0000 7efe5dfeb640 1 librados: init done 2024-03-25T19:30:04.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.483+0000 7efe5dfeb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:04.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.483+0000 7efe5dfeb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:04.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.483+0000 7efe5dfeb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:04.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.483+0000 7efe5dfeb640 1 -- 172.21.15.19:0/1791964115 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe580627b0 con 0x7efe580a98f0 2024-03-25T19:30:04.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.758+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe480035f0 con 0x7efe580a98f0 2024-03-25T19:30:04.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.758+0000 7efe3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:04.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.758+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efe48003900 con 0x7efe580a98f0 2024-03-25T19:30:04.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.758+0000 7efe3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.758+0000 7efe3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.758+0000 7efe3ffff640 10 monclient: dump: 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:04.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.758+0000 7efe55ffb640 4 set_mon_vals no callback set 2024-03-25T19:30:04.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.759+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7efe48019950 con 0x7efe580a98f0 2024-03-25T19:30:04.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.759+0000 7efe56ffd640 10 monclient: get_auth_request con 0x7efe4406df10 auth_method 0 2024-03-25T19:30:04.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.760+0000 7efe3ffff640 10 client.4270.objecter ms_handle_connect 0x7efe4406df10 2024-03-25T19:30:04.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.767+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7efe48053590 con 0x7efe580a98f0 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.767+0000 7efe3ffff640 10 client.4270.objecter ms_dispatch 0x7efe58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.767+0000 7efe3ffff640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.767+0000 7efe3ffff640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.767+0000 7efe3ffff640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T19:30:04.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.768+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe4804eed0 con 0x7efe580a98f0 2024-03-25T19:30:04.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.768+0000 7efe3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:04.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.768+0000 7efe3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:04.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.988+0000 7efe5dfeb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:04.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.988+0000 7efe5dfeb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:04.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.988+0000 7efe5dfeb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:04.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:04.988+0000 7efe5dfeb640 1 -- 172.21.15.19:0/1791964115 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe5813dfd0 con 0x7efe580a98f0 2024-03-25T19:30:05.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.010+0000 7efe3ffff640 1 -- 172.21.15.19:0/1791964115 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7efe48050fd0 con 0x7efe580a98f0 2024-03-25T19:30:05.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.010+0000 7efe3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:05.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.010+0000 7efe3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:05.010 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:30:05.010 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:05.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 10 librados: watch_flush enter 2024-03-25T19:30:05.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 10 librados: watch_flush exit 2024-03-25T19:30:05.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T19:30:05.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 10 client.4270.objecter successfully canceled tick 2024-03-25T19:30:05.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 1 -- 172.21.15.19:0/1791964115 >> v1:172.21.15.188:6800/2207506229 conn(0x7efe4406df10 legacy=0x7efe440703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:05.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 10 monclient: shutdown 2024-03-25T19:30:05.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:05.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 1 -- 172.21.15.19:0/1791964115 >> v1:172.21.15.188:6789/0 conn(0x7efe580a98f0 legacy=0x7efe58132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:05.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 1 -- 172.21.15.19:0/1791964115 shutdown_connections 2024-03-25T19:30:05.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 1 -- 172.21.15.19:0/1791964115 >> 172.21.15.19:0/1791964115 conn(0x7efe580b7ea0 msgr2=0x7efe58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:05.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 1 -- 172.21.15.19:0/1791964115 shutdown_connections 2024-03-25T19:30:05.014 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 1 -- 172.21.15.19:0/1791964115 wait complete. 2024-03-25T19:30:05.014 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:05.012+0000 7efe5dfeb640 1 librados: shutdown 2024-03-25T19:30:05.024 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:11.829 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:34.559 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:11.831 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:11.846 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:34.576 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:11.892 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:01:34.594 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:11.897 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:01:34.600 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:11.907 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:34.636 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:11.912 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:34.641 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:11.916 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:34.646 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:11.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.934+0000 7f9fcd5ed640 10 monclient: get_monmap_and_config 2024-03-25T19:30:11.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.934+0000 7f9fcd5ed640 10 monclient: build_initial_monmap 2024-03-25T19:30:11.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.934+0000 7f9fcd5ed640 10 monclient: monmap: 2024-03-25T19:30:11.934 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:11.934 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:11.935 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:30:11.936080+0000 2024-03-25T19:30:11.935 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:30:11.936080+0000 2024-03-25T19:30:11.935 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:11.935 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:11.935 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:11.935 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:30:11.935 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:30:11.935 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:11.936 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:01:34.638 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:11.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.935+0000 7f9fcd5ed640 1 Processor -- start 2024-03-25T19:30:11.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.935+0000 7f9fcd5ed640 1 -- start start 2024-03-25T19:30:11.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.935+0000 7f9fcd5ed640 10 monclient: init 2024-03-25T19:30:11.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.936+0000 7f9fcd5ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:11.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.936+0000 7f9fcd5ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:11.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.936+0000 7f9fcd5ed640 10 monclient(hunting): picked mon.noname-b con 0x7f9fc813f990 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:11.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.936+0000 7f9fcd5ed640 10 monclient(hunting): picked mon.noname-a con 0x7f9fc813fd70 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:11.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.936+0000 7f9fcd5ed640 10 monclient(hunting): picked mon.noname-c con 0x7f9fc80afa60 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:11.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.936+0000 7f9fcd5ed640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc80afeb0 con 0x7f9fc813fd70 2024-03-25T19:30:11.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.936+0000 7f9fcd5ed640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc80b0100 con 0x7f9fc80afa60 2024-03-25T19:30:11.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.936+0000 7f9fcd5ed640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc80b1150 con 0x7f9fc813f990 2024-03-25T19:30:11.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.936+0000 7f9fcd5ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:11.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.936+0000 7f9fcd5ed640 10 monclient(hunting): authenticate will time out at 6065.206592s 2024-03-25T19:30:11.961 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:34.691 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:11.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.970+0000 7f9fbffff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f9fc813fd70 0x7f9fc8145850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:49678/0 (socket says 172.21.15.19:49678) 2024-03-25T19:30:11.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:11.970+0000 7f9fbffff640 1 -- 172.21.15.19:0/36598343 learned_addr learned my addr 172.21.15.19:0/36598343 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:12.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.004+0000 7f9fc67fc640 1 -- 172.21.15.19:0/36598343 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374812581 0 0) 0x7f9fc80afeb0 con 0x7f9fc813fd70 2024-03-25T19:30:12.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.004+0000 7f9fc67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:12.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.004+0000 7f9fc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:12.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.004+0000 7f9fc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:12.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.004+0000 7f9fc67fc640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:30:12.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.004+0000 7f9fc67fc640 1 -- 172.21.15.19:0/36598343 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fb4003520 con 0x7f9fc813fd70 2024-03-25T19:30:12.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 1 -- 172.21.15.19:0/36598343 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3928005517 0 0) 0x7f9fb4003520 con 0x7f9fc813fd70 2024-03-25T19:30:12.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 1 -- 172.21.15.19:0/36598343 >> v1:172.21.15.19:6790/0 conn(0x7f9fc813f990 legacy=0x7f9fc806ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:12.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 1 -- 172.21.15.19:0/36598343 >> v1:172.21.15.188:6789/0 conn(0x7f9fc80afa60 legacy=0x7f9fc814dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:12.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:12.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 1 monclient: found mon.noname-a 2024-03-25T19:30:12.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 1 -- 172.21.15.19:0/36598343 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc80aede0 con 0x7f9fc813fd70 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 1 -- 172.21.15.19:0/36598343 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9fb0001690 con 0x7f9fc813fd70 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fcd5ed640 5 monclient: authenticate success, global_id 4278 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fcd5ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fc67fc640 10 monclient: dump: 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:12.095 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:12.096 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:12.096 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:12.096 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:12.096 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:12.096 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:12.096 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:12.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.094+0000 7f9fcd5ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:12.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.108+0000 7f9fc67fc640 1 -- 172.21.15.19:0/36598343 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fb00017f0 con 0x7f9fc813fd70 2024-03-25T19:30:12.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.108+0000 7f9fc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:12.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.108+0000 7f9fc67fc640 1 -- 172.21.15.19:0/36598343 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9fb0001c00 con 0x7f9fc813fd70 2024-03-25T19:30:12.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.108+0000 7f9fcd5ed640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:12.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.108+0000 7f9fcd5ed640 4 set_mon_vals no callback set 2024-03-25T19:30:12.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.108+0000 7f9fc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:12.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.108+0000 7f9fc67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:12.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.108+0000 7f9fc67fc640 10 monclient: dump: 2024-03-25T19:30:12.109 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:12.109 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:12.109 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:12.109 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:12.109 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:12.109 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:12.109 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:12.109 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:12.109 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:12.109 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:12.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.109+0000 7f9fcd5ed640 10 monclient: shutdown 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.109+0000 7f9fcd5ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.109+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/36598343 >> v1:172.21.15.19:6789/0 conn(0x7f9fc813fd70 legacy=0x7f9fc8145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.109+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/36598343 shutdown_connections 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.109+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/36598343 >> 172.21.15.19:0/36598343 conn(0x7f9fc80b4b80 msgr2=0x7f9fc80b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.109+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/36598343 shutdown_connections 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.109+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/36598343 wait complete. 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.110+0000 7f9fcd5ed640 10 monclient: build_initial_monmap 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.110+0000 7f9fcd5ed640 10 monclient: monmap: 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.110+0000 7f9fcd5ed640 1 librados: starting msgr at 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.110+0000 7f9fcd5ed640 1 librados: starting objecter 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 1 Processor -- start 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 1 -- start start 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 1 librados: setting wanted keys 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 1 librados: calling monclient init 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 10 monclient: init 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 10 monclient(hunting): picked mon.noname-c con 0x7f9fc80afa60 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 10 monclient(hunting): picked mon.noname-a con 0x7f9fc813f990 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:12.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 10 monclient(hunting): picked mon.noname-b con 0x7f9fc813fd70 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:12.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc8144540 con 0x7f9fc813f990 2024-03-25T19:30:12.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc81447b0 con 0x7f9fc813fd70 2024-03-25T19:30:12.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc8145040 con 0x7f9fc80afa60 2024-03-25T19:30:12.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:12.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.111+0000 7f9fcd5ed640 10 monclient(hunting): authenticate will time out at 6065.381758s 2024-03-25T19:30:12.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.115+0000 7f9fc77fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f9fc813fd70 0x7f9fc81e2f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:41438/0 (socket says 172.21.15.19:41438) 2024-03-25T19:30:12.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.115+0000 7f9fc77fe640 1 -- 172.21.15.19:0/377623902 learned_addr learned my addr 172.21.15.19:0/377623902 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:12.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.118+0000 7f9fbf7fe640 10 client.?.objecter ms_handle_connect 0x7f9fc813fd70 2024-03-25T19:30:12.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.118+0000 7f9fbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:12.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.122+0000 7f9fbf7fe640 10 client.?.objecter ms_handle_connect 0x7f9fc813f990 2024-03-25T19:30:12.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.122+0000 7f9fbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:12.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.126+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913681155 0 0) 0x7f9fc81447b0 con 0x7f9fc813fd70 2024-03-25T19:30:12.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.126+0000 7f9fbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:12.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.126+0000 7f9fbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:12.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.126+0000 7f9fbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:12.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.126+0000 7f9fbf7fe640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:30:12.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.126+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa8004eb0 con 0x7f9fc813fd70 2024-03-25T19:30:12.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.127+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930831440 0 0) 0x7f9fc8144540 con 0x7f9fc813f990 2024-03-25T19:30:12.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.127+0000 7f9fbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:12.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.127+0000 7f9fbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:12.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.127+0000 7f9fbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:12.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.127+0000 7f9fbf7fe640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:30:12.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.127+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc81447b0 con 0x7f9fc813f990 2024-03-25T19:30:12.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.135+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1283119585 0 0) 0x7f9fc81447b0 con 0x7f9fc813f990 2024-03-25T19:30:12.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.135+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fc8144540 con 0x7f9fc813f990 2024-03-25T19:30:12.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.135+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9fb0001b40 con 0x7f9fc813f990 2024-03-25T19:30:12.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.135+0000 7f9fbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:12.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.135+0000 7f9fbf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:12.136 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.135+0000 7f9fbf7fe640 10 monclient(hunting): dump: 2024-03-25T19:30:12.136 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:12.136 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:12.136 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:12.136 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:12.136 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:12.136 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:12.136 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:12.136 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:12.136 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:12.136 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:12.136 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.135+0000 7f9fbf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:12.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 142816437 0 0) 0x7f9fc8144540 con 0x7f9fc813f990 2024-03-25T19:30:12.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 >> v1:172.21.15.19:6790/0 conn(0x7f9fc80afa60 legacy=0x7f9fc8140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:12.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 >> v1:172.21.15.188:6789/0 conn(0x7f9fc813fd70 legacy=0x7f9fc81e2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:12.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fbf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:12.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fbf7fe640 1 monclient: found mon.a 2024-03-25T19:30:12.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:12.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fbf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:12.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc8145560 con 0x7f9fc813f990 2024-03-25T19:30:12.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fbf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:12.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fbf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:12.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:12.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fcd5ed640 5 monclient: authenticate success, global_id 4281 2024-03-25T19:30:12.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fcd5ed640 10 monclient: _renew_subs 2024-03-25T19:30:12.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fcd5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:12.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/377623902 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc8144bd0 con 0x7f9fc813f990 2024-03-25T19:30:12.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fcd5ed640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:12.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fcd5ed640 10 monclient: _renew_subs 2024-03-25T19:30:12.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fcd5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:12.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/377623902 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc813bb40 con 0x7f9fc813f990 2024-03-25T19:30:12.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.144+0000 7f9fcd5ed640 1 librados: init done 2024-03-25T19:30:12.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.145+0000 7f9fcd5ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:12.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.145+0000 7f9fcd5ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:12.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.145+0000 7f9fcd5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:12.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.145+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/377623902 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc8145750 con 0x7f9fc813f990 2024-03-25T19:30:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.164+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fb0004a90 con 0x7f9fc813f990 2024-03-25T19:30:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.164+0000 7f9fbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.164+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9fb0001d80 con 0x7f9fc813f990 2024-03-25T19:30:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.164+0000 7f9fbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.164+0000 7f9fbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.164+0000 7f9fbf7fe640 10 monclient: dump: 2024-03-25T19:30:12.165 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:12.165 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:12.165 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:12.165 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:12.165 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:12.165 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:12.165 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:12.165 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:12.165 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:12.165 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.164+0000 7f9fc5ffb640 4 set_mon_vals no callback set 2024-03-25T19:30:12.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.165+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f9fb001ac10 con 0x7f9fc813f990 2024-03-25T19:30:12.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.166+0000 7f9fc6ffd640 10 monclient: get_auth_request con 0x7f9fa806dc90 auth_method 0 2024-03-25T19:30:12.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.166+0000 7f9fbf7fe640 10 client.4281.objecter ms_handle_connect 0x7f9fa806dc90 2024-03-25T19:30:12.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.458+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f9fb0054870 con 0x7f9fc813f990 2024-03-25T19:30:12.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.458+0000 7f9fbf7fe640 10 client.4281.objecter ms_dispatch 0x7f9fc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:12.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.458+0000 7f9fbf7fe640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:12.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.458+0000 7f9fbf7fe640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:12.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.458+0000 7f9fbf7fe640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-25T19:30:12.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.458+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9fb00501b0 con 0x7f9fc813f990 2024-03-25T19:30:12.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.458+0000 7f9fbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:12.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.458+0000 7f9fbf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:12.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.716+0000 7f9fcd5ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:12.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.716+0000 7f9fcd5ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:12.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.716+0000 7f9fcd5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:12.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.716+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/377623902 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fc81451d0 con 0x7f9fc813f990 2024-03-25T19:30:12.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.744+0000 7f9fbf7fe640 1 -- 172.21.15.19:0/377623902 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f9fb00522b0 con 0x7f9fc813f990 2024-03-25T19:30:12.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.744+0000 7f9fbf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:12.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.744+0000 7f9fbf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:12.745 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:30:12.745 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:12.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 10 librados: watch_flush enter 2024-03-25T19:30:12.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 10 librados: watch_flush exit 2024-03-25T19:30:12.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-25T19:30:12.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 10 client.4281.objecter successfully canceled tick 2024-03-25T19:30:12.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/377623902 >> v1:172.21.15.188:6800/2207506229 conn(0x7f9fa806dc90 legacy=0x7f9fa8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:12.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 10 monclient: shutdown 2024-03-25T19:30:12.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:12.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/377623902 >> v1:172.21.15.19:6789/0 conn(0x7f9fc813f990 legacy=0x7f9fc813b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:12.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/377623902 shutdown_connections 2024-03-25T19:30:12.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/377623902 >> 172.21.15.19:0/377623902 conn(0x7f9fc80b4b80 msgr2=0x7f9fc80b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:12.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/377623902 shutdown_connections 2024-03-25T19:30:12.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 1 -- 172.21.15.19:0/377623902 wait complete. 2024-03-25T19:30:12.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:12.747+0000 7f9fcd5ed640 1 librados: shutdown 2024-03-25T19:30:12.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:19.567 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:19.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.684+0000 7fa2b39e9640 10 monclient: get_monmap_and_config 2024-03-25T19:30:19.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.684+0000 7fa2b39e9640 10 monclient: build_initial_monmap 2024-03-25T19:30:19.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.685+0000 7fa2b39e9640 10 monclient: monmap: 2024-03-25T19:30:19.684 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:19.684 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:19.684 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:30:19.685246+0000 2024-03-25T19:30:19.684 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:30:19.685246+0000 2024-03-25T19:30:19.684 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:19.684 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:19.684 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:19.684 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:30:19.684 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:30:19.685 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:19.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.685+0000 7fa2b39e9640 1 Processor -- start 2024-03-25T19:30:19.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.686+0000 7fa2b39e9640 1 -- start start 2024-03-25T19:30:19.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.686+0000 7fa2b39e9640 10 monclient: init 2024-03-25T19:30:19.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.686+0000 7fa2b39e9640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:19.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.686+0000 7fa2b39e9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:19.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.686+0000 7fa2b39e9640 10 monclient(hunting): picked mon.noname-a con 0x7fa2ac13f990 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:19.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.686+0000 7fa2b39e9640 10 monclient(hunting): picked mon.noname-c con 0x7fa2ac13fd70 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:19.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.686+0000 7fa2b39e9640 10 monclient(hunting): picked mon.noname-b con 0x7fa2ac0afa60 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:19.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.686+0000 7fa2b39e9640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac0afeb0 con 0x7fa2ac13f990 2024-03-25T19:30:19.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.686+0000 7fa2b39e9640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac0b0100 con 0x7fa2ac13fd70 2024-03-25T19:30:19.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.686+0000 7fa2b39e9640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac0b1150 con 0x7fa2ac0afa60 2024-03-25T19:30:19.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.686+0000 7fa2b39e9640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:19.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.686+0000 7fa2b39e9640 10 monclient(hunting): authenticate will time out at 6072.955886s 2024-03-25T19:30:19.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.695+0000 7fa2b0f84640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fa2ac13fd70 0x7fa2ac145850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:51902/0 (socket says 172.21.15.19:51902) 2024-03-25T19:30:19.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.695+0000 7fa2b0f84640 1 -- 172.21.15.19:0/2572794623 learned_addr learned my addr 172.21.15.19:0/2572794623 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:19.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.708+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161529256 0 0) 0x7fa2ac0b1150 con 0x7fa2ac0afa60 2024-03-25T19:30:19.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.708+0000 7fa2a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.708+0000 7fa2a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.708+0000 7fa2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.708+0000 7fa2a3fff640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:30:19.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.708+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa298003520 con 0x7fa2ac0afa60 2024-03-25T19:30:19.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.712+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385707225 0 0) 0x7fa2ac0b0100 con 0x7fa2ac13fd70 2024-03-25T19:30:19.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.712+0000 7fa2a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.712+0000 7fa2a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.712+0000 7fa2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.712+0000 7fa2a3fff640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:30:19.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.712+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ac0b1150 con 0x7fa2ac13fd70 2024-03-25T19:30:19.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.714+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286572447 0 0) 0x7fa2ac0afeb0 con 0x7fa2ac13f990 2024-03-25T19:30:19.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.714+0000 7fa2a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.714+0000 7fa2a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.714+0000 7fa2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.714+0000 7fa2a3fff640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:30:19.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.714+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ac0b0100 con 0x7fa2ac13f990 2024-03-25T19:30:19.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 835264814 0 0) 0x7fa298003520 con 0x7fa2ac0afa60 2024-03-25T19:30:19.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 >> v1:172.21.15.188:6789/0 conn(0x7fa2ac13fd70 legacy=0x7fa2ac145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 >> v1:172.21.15.19:6789/0 conn(0x7fa2ac13f990 legacy=0x7fa2ac06ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:19.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 1 monclient: found mon.noname-b 2024-03-25T19:30:19.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:19.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:30:19.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac0aede0 con 0x7fa2ac0afa60 2024-03-25T19:30:19.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 10 monclient: _finish_auth 0 2024-03-25T19:30:19.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa2a4001a10 con 0x7fa2ac0afa60 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2b39e9640 5 monclient: authenticate success, global_id 4262 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2b39e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.725+0000 7fa2a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.726+0000 7fa2a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.726+0000 7fa2a3fff640 10 monclient: dump: 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.726 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:19.727 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:19.727 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:19.727 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:19.727 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:19.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.726+0000 7fa2b39e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:19.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.738+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2a4001b70 con 0x7fa2ac0afa60 2024-03-25T19:30:19.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.738+0000 7fa2a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:19.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.738+0000 7fa2a3fff640 1 -- 172.21.15.19:0/2572794623 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa2a4001f80 con 0x7fa2ac0afa60 2024-03-25T19:30:19.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.738+0000 7fa2a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.738+0000 7fa2a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:19.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.738+0000 7fa2a3fff640 10 monclient: dump: 2024-03-25T19:30:19.738 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:19.738 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:19.738 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:19.738 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:19.738 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.738 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:19.738 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:19.738 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:19.738 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:19.738 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:19.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.738+0000 7fa2b39e9640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:19.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.738+0000 7fa2b39e9640 4 set_mon_vals no callback set 2024-03-25T19:30:19.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.739+0000 7fa2b39e9640 10 monclient: shutdown 2024-03-25T19:30:19.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.739+0000 7fa2b39e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.739+0000 7fa2b39e9640 1 -- 172.21.15.19:0/2572794623 >> v1:172.21.15.19:6790/0 conn(0x7fa2ac0afa60 legacy=0x7fa2ac14dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.739+0000 7fa2b39e9640 1 -- 172.21.15.19:0/2572794623 shutdown_connections 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.739+0000 7fa2b39e9640 1 -- 172.21.15.19:0/2572794623 >> 172.21.15.19:0/2572794623 conn(0x7fa2ac0b4b80 msgr2=0x7fa2ac0b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.739+0000 7fa2b39e9640 1 -- 172.21.15.19:0/2572794623 shutdown_connections 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.739+0000 7fa2b39e9640 1 -- 172.21.15.19:0/2572794623 wait complete. 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.740+0000 7fa2b39e9640 10 monclient: build_initial_monmap 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.740+0000 7fa2b39e9640 10 monclient: monmap: 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:19.741 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.740+0000 7fa2b39e9640 1 librados: starting msgr at 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.740+0000 7fa2b39e9640 1 librados: starting objecter 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.740+0000 7fa2b39e9640 1 Processor -- start 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.740+0000 7fa2b39e9640 1 -- start start 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.740+0000 7fa2b39e9640 1 librados: setting wanted keys 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.740+0000 7fa2b39e9640 1 librados: calling monclient init 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.740+0000 7fa2b39e9640 10 monclient: init 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.741+0000 7fa2b39e9640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.741+0000 7fa2b39e9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:19.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.741+0000 7fa2b39e9640 10 monclient(hunting): picked mon.noname-b con 0x7fa2ac0afa60 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:19.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.741+0000 7fa2b39e9640 10 monclient(hunting): picked mon.noname-c con 0x7fa2ac13f990 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:19.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.741+0000 7fa2b39e9640 10 monclient(hunting): picked mon.noname-a con 0x7fa2ac13fd70 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:19.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.741+0000 7fa2b39e9640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac1cbed0 con 0x7fa2ac13fd70 2024-03-25T19:30:19.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.741+0000 7fa2b39e9640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac1ce250 con 0x7fa2ac0afa60 2024-03-25T19:30:19.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.741+0000 7fa2b39e9640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac1ce530 con 0x7fa2ac13f990 2024-03-25T19:30:19.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.741+0000 7fa2b39e9640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:19.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.741+0000 7fa2b39e9640 10 monclient(hunting): authenticate will time out at 6073.010758s 2024-03-25T19:30:19.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.751+0000 7fa2b1f86640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fa2ac13fd70 0x7fa2ac1e2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:48492/0 (socket says 172.21.15.19:48492) 2024-03-25T19:30:19.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.751+0000 7fa2b1f86640 1 -- 172.21.15.19:0/1593229944 learned_addr learned my addr 172.21.15.19:0/1593229944 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:19.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.751+0000 7fa2a1ffb640 10 client.?.objecter ms_handle_connect 0x7fa2ac13fd70 2024-03-25T19:30:19.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.751+0000 7fa2a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:19.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.758+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286068261 0 0) 0x7fa2ac1cbed0 con 0x7fa2ac13fd70 2024-03-25T19:30:19.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.758+0000 7fa2a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.758+0000 7fa2a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.758+0000 7fa2a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.758+0000 7fa2a1ffb640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:30:19.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.758+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa290004eb0 con 0x7fa2ac13fd70 2024-03-25T19:30:19.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.761+0000 7fa2a1ffb640 10 client.?.objecter ms_handle_connect 0x7fa2ac0afa60 2024-03-25T19:30:19.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.761+0000 7fa2a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:19.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.766+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4053336639 0 0) 0x7fa290004eb0 con 0x7fa2ac13fd70 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.766+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2ac1cbed0 con 0x7fa2ac13fd70 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.766+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa2a4001ec0 con 0x7fa2ac13fd70 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.766+0000 7fa2a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.766+0000 7fa2a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.766+0000 7fa2a1ffb640 10 monclient(hunting): dump: 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:19.766 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:19.767 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:19.767 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:19.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.766+0000 7fa2a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:19.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.772+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895730753 0 0) 0x7fa2ac1ce250 con 0x7fa2ac0afa60 2024-03-25T19:30:19.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.772+0000 7fa2a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.772+0000 7fa2a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.772+0000 7fa2a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.772+0000 7fa2a1ffb640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:30:19.772 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.772+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa290004eb0 con 0x7fa2ac0afa60 2024-03-25T19:30:19.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3254321411 0 0) 0x7fa2ac1cbed0 con 0x7fa2ac13fd70 2024-03-25T19:30:19.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 >> v1:172.21.15.19:6790/0 conn(0x7fa2ac13f990 legacy=0x7fa2ac1d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 >> v1:172.21.15.188:6789/0 conn(0x7fa2ac0afa60 legacy=0x7fa2ac1cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2a1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2a1ffb640 1 monclient: found mon.a 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2a1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac1cf3d0 con 0x7fa2ac13fd70 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2a1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2a1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2b39e9640 5 monclient: authenticate success, global_id 4287 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2b39e9640 10 monclient: _renew_subs 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2b39e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2b39e9640 1 -- 172.21.15.19:0/1593229944 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ac13bed0 con 0x7fa2ac13fd70 2024-03-25T19:30:19.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2b39e9640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:19.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2b39e9640 10 monclient: _renew_subs 2024-03-25T19:30:19.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2b39e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:19.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2b39e9640 1 -- 172.21.15.19:0/1593229944 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ac1d8620 con 0x7fa2ac13fd70 2024-03-25T19:30:19.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.776+0000 7fa2b39e9640 1 librados: init done 2024-03-25T19:30:19.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.777+0000 7fa2b39e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:19.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.777+0000 7fa2b39e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:19.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.777+0000 7fa2b39e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:19.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:19.777+0000 7fa2b39e9640 1 -- 172.21.15.19:0/1593229944 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ac145ee0 con 0x7fa2ac13fd70 2024-03-25T19:30:20.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.092+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2a4003c20 con 0x7fa2ac13fd70 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.092+0000 7fa2a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.092+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa2a4004070 con 0x7fa2ac13fd70 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.092+0000 7fa2a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.092+0000 7fa2a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.092+0000 7fa2a1ffb640 10 monclient: dump: 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:20.092 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:20.093 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:20.093 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:20.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.092+0000 7fa2a37fe640 4 set_mon_vals no callback set 2024-03-25T19:30:20.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.093+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fa2a4004290 con 0x7fa2ac13fd70 2024-03-25T19:30:20.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.094+0000 7fa2b1785640 10 monclient: get_auth_request con 0x7fa29006ddc0 auth_method 0 2024-03-25T19:30:20.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.094+0000 7fa2a1ffb640 10 client.4287.objecter ms_handle_connect 0x7fa29006ddc0 2024-03-25T19:30:20.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.099+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fa2a4054c80 con 0x7fa2ac13fd70 2024-03-25T19:30:20.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.099+0000 7fa2a1ffb640 10 client.4287.objecter ms_dispatch 0x7fa2ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:20.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.099+0000 7fa2a1ffb640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:20.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.099+0000 7fa2a1ffb640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:20.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.099+0000 7fa2a1ffb640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-25T19:30:20.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.099+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2a4050790 con 0x7fa2ac13fd70 2024-03-25T19:30:20.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.099+0000 7fa2a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:20.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.099+0000 7fa2a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:20.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.310+0000 7fa2b39e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:20.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.310+0000 7fa2b39e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:20.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.310+0000 7fa2b39e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:20.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.310+0000 7fa2b39e9640 1 -- 172.21.15.19:0/1593229944 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2ac0627b0 con 0x7fa2ac13fd70 2024-03-25T19:30:20.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.348+0000 7fa2a1ffb640 1 -- 172.21.15.19:0/1593229944 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fa2a40526c0 con 0x7fa2ac13fd70 2024-03-25T19:30:20.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.348+0000 7fa2a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:20.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.348+0000 7fa2a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:20.347 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:30:20.347 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:20.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.350+0000 7fa2b39e9640 10 librados: watch_flush enter 2024-03-25T19:30:20.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.350+0000 7fa2b39e9640 10 librados: watch_flush exit 2024-03-25T19:30:20.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.350+0000 7fa2b39e9640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-25T19:30:20.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.350+0000 7fa2b39e9640 10 client.4287.objecter successfully canceled tick 2024-03-25T19:30:20.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.351+0000 7fa2b39e9640 1 -- 172.21.15.19:0/1593229944 >> v1:172.21.15.188:6800/2207506229 conn(0x7fa29006ddc0 legacy=0x7fa290070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:20.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.351+0000 7fa2b39e9640 10 monclient: shutdown 2024-03-25T19:30:20.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.351+0000 7fa2b39e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:20.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.351+0000 7fa2b39e9640 1 -- 172.21.15.19:0/1593229944 >> v1:172.21.15.19:6789/0 conn(0x7fa2ac13fd70 legacy=0x7fa2ac1e2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:20.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.351+0000 7fa2b39e9640 1 -- 172.21.15.19:0/1593229944 shutdown_connections 2024-03-25T19:30:20.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.351+0000 7fa2b39e9640 1 -- 172.21.15.19:0/1593229944 >> 172.21.15.19:0/1593229944 conn(0x7fa2ac0b4b80 msgr2=0x7fa2ac0b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:20.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.351+0000 7fa2b39e9640 1 -- 172.21.15.19:0/1593229944 shutdown_connections 2024-03-25T19:30:20.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.351+0000 7fa2b39e9640 1 -- 172.21.15.19:0/1593229944 wait complete. 2024-03-25T19:30:20.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:20.351+0000 7fa2b39e9640 1 librados: shutdown 2024-03-25T19:30:20.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:27.169 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:27.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.286+0000 7f4f93df6640 10 monclient: get_monmap_and_config 2024-03-25T19:30:27.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.286+0000 7f4f93df6640 10 monclient: build_initial_monmap 2024-03-25T19:30:27.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.286+0000 7f4f93df6640 10 monclient: monmap: 2024-03-25T19:30:27.285 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:27.285 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:27.285 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:30:27.286467+0000 2024-03-25T19:30:27.285 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:30:27.286467+0000 2024-03-25T19:30:27.286 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:27.286 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:27.286 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:27.286 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:30:27.286 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:30:27.286 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:27.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.287+0000 7f4f93df6640 1 Processor -- start 2024-03-25T19:30:27.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.287+0000 7f4f93df6640 1 -- start start 2024-03-25T19:30:27.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.287+0000 7f4f93df6640 10 monclient: init 2024-03-25T19:30:27.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.288+0000 7f4f93df6640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:27.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.288+0000 7f4f93df6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:27.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.288+0000 7f4f93df6640 10 monclient(hunting): picked mon.noname-b con 0x7f4f8c0a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:27.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.288+0000 7f4f93df6640 10 monclient(hunting): picked mon.noname-c con 0x7f4f8c13f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:27.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.288+0000 7f4f93df6640 10 monclient(hunting): picked mon.noname-a con 0x7f4f8c1430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:27.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.288+0000 7f4f93df6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c142520 con 0x7f4f8c1430c0 2024-03-25T19:30:27.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.288+0000 7f4f93df6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c0b28a0 con 0x7f4f8c13f710 2024-03-25T19:30:27.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.288+0000 7f4f93df6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c0b2a30 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.288+0000 7f4f93df6640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:27.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.288+0000 7f4f93df6640 10 monclient(hunting): authenticate will time out at 6080.557226s 2024-03-25T19:30:27.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.290+0000 7f4f91b92640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f4f8c0a98f0 0x7f4f8c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:60266/0 (socket says 172.21.15.19:60266) 2024-03-25T19:30:27.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.290+0000 7f4f91b92640 1 -- 172.21.15.19:0/799761148 learned_addr learned my addr 172.21.15.19:0/799761148 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:27.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.299+0000 7f4f90b90640 1 -- 172.21.15.19:0/799761148 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346663906 0 0) 0x7f4f8c0b2a30 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.299+0000 7f4f90b90640 10 monclient(hunting): _negotiate 2024-03-25T19:30:27.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.299+0000 7f4f90b90640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:27.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.299+0000 7f4f90b90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:27.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.299+0000 7f4f90b90640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:30:27.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.299+0000 7f4f90b90640 1 -- 172.21.15.19:0/799761148 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f78003520 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 1 -- 172.21.15.19:0/799761148 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3221129418 0 0) 0x7f4f78003520 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 1 -- 172.21.15.19:0/799761148 >> v1:172.21.15.188:6789/0 conn(0x7f4f8c13f710 legacy=0x7f4f8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 1 -- 172.21.15.19:0/799761148 >> v1:172.21.15.19:6789/0 conn(0x7f4f8c1430c0 legacy=0x7f4f8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 10 monclient: _finish_hunting 0 2024-03-25T19:30:27.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 1 monclient: found mon.noname-b 2024-03-25T19:30:27.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:27.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:30:27.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 1 -- 172.21.15.19:0/799761148 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c0b44c0 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 10 monclient: _finish_auth 0 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 10 monclient: _check_auth_tickets 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 1 -- 172.21.15.19:0/799761148 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f4f88001690 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.313+0000 7f4f90b90640 10 monclient: dump: 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:27.314 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:27.315 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:27.315 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:27.315 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:27.315 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:27.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.314+0000 7f4f93df6640 5 monclient: authenticate success, global_id 4265 2024-03-25T19:30:27.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.314+0000 7f4f93df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:27.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.320+0000 7f4f90b90640 1 -- 172.21.15.19:0/799761148 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f880017f0 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.320+0000 7f4f90b90640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:27.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.320+0000 7f4f90b90640 1 -- 172.21.15.19:0/799761148 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f4f88001c00 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.320+0000 7f4f90b90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:27.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.320+0000 7f4f90b90640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:27.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.320+0000 7f4f90b90640 10 monclient: dump: 2024-03-25T19:30:27.319 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:27.319 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:27.319 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:27.319 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:27.320 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:27.320 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:27.320 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:27.320 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:27.320 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:27.320 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:27.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.320+0000 7f4f93df6640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:27.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.320+0000 7f4f93df6640 4 set_mon_vals no callback set 2024-03-25T19:30:27.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.321+0000 7f4f93df6640 10 monclient: shutdown 2024-03-25T19:30:27.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.321+0000 7f4f93df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:27.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.321+0000 7f4f93df6640 1 -- 172.21.15.19:0/799761148 >> v1:172.21.15.19:6790/0 conn(0x7f4f8c0a98f0 legacy=0x7f4f8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.321+0000 7f4f93df6640 1 -- 172.21.15.19:0/799761148 shutdown_connections 2024-03-25T19:30:27.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.321+0000 7f4f93df6640 1 -- 172.21.15.19:0/799761148 >> 172.21.15.19:0/799761148 conn(0x7f4f8c0b7ea0 msgr2=0x7f4f8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:27.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.321+0000 7f4f93df6640 1 -- 172.21.15.19:0/799761148 shutdown_connections 2024-03-25T19:30:27.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.321+0000 7f4f93df6640 1 -- 172.21.15.19:0/799761148 wait complete. 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.322+0000 7f4f93df6640 10 monclient: build_initial_monmap 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.322+0000 7f4f93df6640 10 monclient: monmap: 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.322+0000 7f4f93df6640 1 librados: starting msgr at 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.322+0000 7f4f93df6640 1 librados: starting objecter 2024-03-25T19:30:27.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.322+0000 7f4f93df6640 1 Processor -- start 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.322+0000 7f4f93df6640 1 -- start start 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.322+0000 7f4f93df6640 1 librados: setting wanted keys 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.322+0000 7f4f93df6640 1 librados: calling monclient init 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.322+0000 7f4f93df6640 10 monclient: init 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.323+0000 7f4f93df6640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.323+0000 7f4f93df6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.323+0000 7f4f93df6640 10 monclient(hunting): picked mon.noname-a con 0x7f4f8c0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.323+0000 7f4f93df6640 10 monclient(hunting): picked mon.noname-b con 0x7f4f8c13f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.323+0000 7f4f93df6640 10 monclient(hunting): picked mon.noname-c con 0x7f4f8c1430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.323+0000 7f4f93df6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c135730 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.323+0000 7f4f93df6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c1358c0 con 0x7f4f8c13f710 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.323+0000 7f4f93df6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c135a50 con 0x7f4f8c1430c0 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.323+0000 7f4f93df6640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:27.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.323+0000 7f4f93df6640 10 monclient(hunting): authenticate will time out at 6080.592510s 2024-03-25T19:30:27.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.335+0000 7f4f91391640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f4f8c13f710 0x7f4f8c1d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:56698/0 (socket says 172.21.15.19:56698) 2024-03-25T19:30:27.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.335+0000 7f4f91391640 1 -- 172.21.15.19:0/2486795375 learned_addr learned my addr 172.21.15.19:0/2486795375 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:27.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.336+0000 7f4f827fc640 10 client.?.objecter ms_handle_connect 0x7f4f8c13f710 2024-03-25T19:30:27.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.336+0000 7f4f827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:27.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.340+0000 7f4f827fc640 10 client.?.objecter ms_handle_connect 0x7f4f8c1430c0 2024-03-25T19:30:27.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.340+0000 7f4f827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:27.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.346+0000 7f4f827fc640 10 client.?.objecter ms_handle_connect 0x7f4f8c0a98f0 2024-03-25T19:30:27.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.346+0000 7f4f827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:27.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.347+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473615689 0 0) 0x7f4f8c1358c0 con 0x7f4f8c13f710 2024-03-25T19:30:27.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.347+0000 7f4f827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:27.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.347+0000 7f4f827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:27.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.347+0000 7f4f827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:27.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.347+0000 7f4f827fc640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:30:27.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.347+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f70004eb0 con 0x7f4f8c13f710 2024-03-25T19:30:27.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.355+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171309506 0 0) 0x7f4f8c135730 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.355+0000 7f4f827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:27.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.355+0000 7f4f827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:27.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.355+0000 7f4f827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:27.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.355+0000 7f4f827fc640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:30:27.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.355+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f8c1358c0 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.360+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1200889342 0 0) 0x7f4f70004eb0 con 0x7f4f8c13f710 2024-03-25T19:30:27.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.360+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f8c135730 con 0x7f4f8c13f710 2024-03-25T19:30:27.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.360+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f4f7c001a30 con 0x7f4f8c13f710 2024-03-25T19:30:27.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.360+0000 7f4f827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:27.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.360+0000 7f4f827fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:27.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.360+0000 7f4f827fc640 10 monclient(hunting): dump: 2024-03-25T19:30:27.360 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:27.360 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:27.360 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:27.360 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:27.360 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:27.360 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:27.360 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:27.360 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:27.360 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:27.360 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:27.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.360+0000 7f4f827fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:27.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.363+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326537564 0 0) 0x7f4f8c135a50 con 0x7f4f8c1430c0 2024-03-25T19:30:27.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.363+0000 7f4f827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:27.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.363+0000 7f4f827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:27.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.363+0000 7f4f827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:27.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.363+0000 7f4f827fc640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:30:27.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.363+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f70004eb0 con 0x7f4f8c1430c0 2024-03-25T19:30:27.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.368+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1491874484 0 0) 0x7f4f8c1358c0 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.368+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f8c135a50 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.368+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f4f88001b00 con 0x7f4f8c0a98f0 2024-03-25T19:30:27.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.368+0000 7f4f827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:27.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.368+0000 7f4f827fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:27.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.368+0000 7f4f827fc640 10 monclient(hunting): dump: 2024-03-25T19:30:27.367 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:27.367 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:27.367 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:27.367 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:27.368 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:27.368 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:27.368 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:27.368 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:27.368 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:27.368 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:27.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.369+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3588756933 0 0) 0x7f4f8c135730 con 0x7f4f8c13f710 2024-03-25T19:30:27.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.369+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 >> v1:172.21.15.19:6790/0 conn(0x7f4f8c1430c0 legacy=0x7f4f8c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.369+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 >> v1:172.21.15.19:6789/0 conn(0x7f4f8c0a98f0 legacy=0x7f4f8c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.369+0000 7f4f827fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:27.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.369+0000 7f4f827fc640 1 monclient: found mon.b 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.369+0000 7f4f827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.369+0000 7f4f827fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.369+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c1d5000 con 0x7f4f8c13f710 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.369+0000 7f4f827fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.369+0000 7f4f827fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.369+0000 7f4f827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.369+0000 7f4f93df6640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.370+0000 7f4f93df6640 10 monclient: _renew_subs 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.370+0000 7f4f93df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.370+0000 7f4f93df6640 1 -- 172.21.15.19:0/2486795375 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8c1d52b0 con 0x7f4f8c13f710 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.370+0000 7f4f93df6640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.370+0000 7f4f93df6640 10 monclient: _renew_subs 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.370+0000 7f4f93df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:27.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.370+0000 7f4f93df6640 1 -- 172.21.15.19:0/2486795375 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f8c13c900 con 0x7f4f8c13f710 2024-03-25T19:30:27.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.370+0000 7f4f93df6640 1 librados: init done 2024-03-25T19:30:27.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.371+0000 7f4f93df6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:27.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.371+0000 7f4f93df6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:27.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.371+0000 7f4f93df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:27.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.371+0000 7f4f93df6640 1 -- 172.21.15.19:0/2486795375 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8c1371c0 con 0x7f4f8c13f710 2024-03-25T19:30:27.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.652+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f7c001c70 con 0x7f4f8c13f710 2024-03-25T19:30:27.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.652+0000 7f4f827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:27.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.652+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f4f7c003a60 con 0x7f4f8c13f710 2024-03-25T19:30:27.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.652+0000 7f4f827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:27.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.652+0000 7f4f827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:27.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.652+0000 7f4f827fc640 10 monclient: dump: 2024-03-25T19:30:27.652 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:27.652 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:27.652 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:27.652 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:27.652 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:27.652 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:27.652 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:27.653 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:27.653 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:27.653 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:27.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.652+0000 7f4f83fff640 4 set_mon_vals no callback set 2024-03-25T19:30:27.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.653+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f4f7c003c80 con 0x7f4f8c13f710 2024-03-25T19:30:27.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.654+0000 7f4f91b92640 10 monclient: get_auth_request con 0x7f4f7006dfb0 auth_method 0 2024-03-25T19:30:27.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.654+0000 7f4f827fc640 10 client.4288.objecter ms_handle_connect 0x7f4f7006dfb0 2024-03-25T19:30:27.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.658+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f4f7c054850 con 0x7f4f8c13f710 2024-03-25T19:30:27.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.658+0000 7f4f827fc640 10 client.4288.objecter ms_dispatch 0x7f4f8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:27.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.658+0000 7f4f827fc640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:27.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.658+0000 7f4f827fc640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:27.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.658+0000 7f4f827fc640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T19:30:27.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.659+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f7c050360 con 0x7f4f8c13f710 2024-03-25T19:30:27.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.659+0000 7f4f827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:27.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.659+0000 7f4f827fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:27.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.919+0000 7f4f93df6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:27.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.919+0000 7f4f93df6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:27.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.919+0000 7f4f93df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:27.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.919+0000 7f4f93df6640 1 -- 172.21.15.19:0/2486795375 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f8c0627b0 con 0x7f4f8c13f710 2024-03-25T19:30:27.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.955+0000 7f4f827fc640 1 -- 172.21.15.19:0/2486795375 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f4f7c052290 con 0x7f4f8c13f710 2024-03-25T19:30:27.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.955+0000 7f4f827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:27.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.955+0000 7f4f827fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:27.954 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:30:27.955 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:27.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 10 librados: watch_flush enter 2024-03-25T19:30:27.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 10 librados: watch_flush exit 2024-03-25T19:30:27.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T19:30:27.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 10 client.4288.objecter successfully canceled tick 2024-03-25T19:30:27.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 1 -- 172.21.15.19:0/2486795375 >> v1:172.21.15.188:6800/2207506229 conn(0x7f4f7006dfb0 legacy=0x7f4f70070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 10 monclient: shutdown 2024-03-25T19:30:27.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:27.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 1 -- 172.21.15.19:0/2486795375 >> v1:172.21.15.188:6789/0 conn(0x7f4f8c13f710 legacy=0x7f4f8c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 1 -- 172.21.15.19:0/2486795375 shutdown_connections 2024-03-25T19:30:27.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 1 -- 172.21.15.19:0/2486795375 >> 172.21.15.19:0/2486795375 conn(0x7f4f8c0b7ea0 msgr2=0x7f4f8c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:27.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 1 -- 172.21.15.19:0/2486795375 shutdown_connections 2024-03-25T19:30:27.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 1 -- 172.21.15.19:0/2486795375 wait complete. 2024-03-25T19:30:27.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:27.957+0000 7f4f93df6640 1 librados: shutdown 2024-03-25T19:30:27.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:31.976 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:54.705 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:31.980 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:54.709 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:32.051 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:54.781 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:32.081 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:54.811 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:32.090 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:54.820 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:32.107 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:01:54.810 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:32.113 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:01:54.816 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:32.127 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:54.856 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:32.139 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:01:54.842 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:34.775 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:34.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.882+0000 7f5274a8d640 10 monclient: get_monmap_and_config 2024-03-25T19:30:34.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.882+0000 7f5274a8d640 10 monclient: build_initial_monmap 2024-03-25T19:30:34.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.882+0000 7f5274a8d640 10 monclient: monmap: 2024-03-25T19:30:34.882 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:34.882 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:34.882 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:30:34.882825+0000 2024-03-25T19:30:34.882 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:30:34.882825+0000 2024-03-25T19:30:34.882 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:34.882 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:34.882 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:34.882 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:30:34.882 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:30:34.882 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:34.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.883+0000 7f5274a8d640 1 Processor -- start 2024-03-25T19:30:34.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.883+0000 7f5274a8d640 1 -- start start 2024-03-25T19:30:34.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.883+0000 7f5274a8d640 10 monclient: init 2024-03-25T19:30:34.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.884+0000 7f5274a8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:34.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.884+0000 7f5274a8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:34.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.884+0000 7f5274a8d640 10 monclient(hunting): picked mon.noname-c con 0x7f527013fc60 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:34.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.884+0000 7f5274a8d640 10 monclient(hunting): picked mon.noname-b con 0x7f5270070570 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:34.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.884+0000 7f5274a8d640 10 monclient(hunting): picked mon.noname-a con 0x7f52700727a0 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:34.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.884+0000 7f5274a8d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527013c9f0 con 0x7f52700727a0 2024-03-25T19:30:34.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.884+0000 7f5274a8d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527006fac0 con 0x7f527013fc60 2024-03-25T19:30:34.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.884+0000 7f5274a8d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52700715d0 con 0x7f5270070570 2024-03-25T19:30:34.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.884+0000 7f5274a8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:34.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.884+0000 7f5274a8d640 10 monclient(hunting): authenticate will time out at 6088.153380s 2024-03-25T19:30:34.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.887+0000 7f526dd9b640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f5270070570 0x7f5270072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:34144/0 (socket says 172.21.15.19:34144) 2024-03-25T19:30:34.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.887+0000 7f526dd9b640 1 -- 172.21.15.19:0/238228004 learned_addr learned my addr 172.21.15.19:0/238228004 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:34.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.896+0000 7f526d59a640 1 -- 172.21.15.19:0/238228004 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949629606 0 0) 0x7f52700715d0 con 0x7f5270070570 2024-03-25T19:30:34.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.896+0000 7f526d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.896+0000 7f526d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.896+0000 7f526d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.896+0000 7f526d59a640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:30:34.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.896+0000 7f526d59a640 1 -- 172.21.15.19:0/238228004 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5258003520 con 0x7f5270070570 2024-03-25T19:30:34.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.906+0000 7f526d59a640 1 -- 172.21.15.19:0/238228004 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780696717 0 0) 0x7f527006fac0 con 0x7f527013fc60 2024-03-25T19:30:34.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.906+0000 7f526d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.906+0000 7f526d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.906 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.906+0000 7f526d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.906 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.906+0000 7f526d59a640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:30:34.906 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.906+0000 7f526d59a640 1 -- 172.21.15.19:0/238228004 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52700715d0 con 0x7f527013fc60 2024-03-25T19:30:34.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 1 -- 172.21.15.19:0/238228004 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1416827475 0 0) 0x7f52700715d0 con 0x7f527013fc60 2024-03-25T19:30:34.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 1 -- 172.21.15.19:0/238228004 >> v1:172.21.15.19:6790/0 conn(0x7f5270070570 legacy=0x7f5270072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 1 -- 172.21.15.19:0/238228004 >> v1:172.21.15.19:6789/0 conn(0x7f52700727a0 legacy=0x7f527014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:30:34.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 1 monclient: found mon.noname-c 2024-03-25T19:30:34.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:34.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:30:34.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 1 -- 172.21.15.19:0/238228004 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52700718b0 con 0x7f527013fc60 2024-03-25T19:30:34.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 10 monclient: _finish_auth 0 2024-03-25T19:30:34.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 1 -- 172.21.15.19:0/238228004 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5260001670 con 0x7f527013fc60 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f5274a8d640 5 monclient: authenticate success, global_id 4291 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f5274a8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f526d59a640 10 monclient: dump: 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:34.920 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:34.921 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:34.921 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:34.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.920+0000 7f5274a8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:34.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.941+0000 7f526d59a640 1 -- 172.21.15.19:0/238228004 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52600017d0 con 0x7f527013fc60 2024-03-25T19:30:34.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.941+0000 7f526d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:34.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.941+0000 7f526d59a640 1 -- 172.21.15.19:0/238228004 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5260001ba0 con 0x7f527013fc60 2024-03-25T19:30:34.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.941+0000 7f5274a8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:34.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.941+0000 7f5274a8d640 4 set_mon_vals no callback set 2024-03-25T19:30:34.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.941+0000 7f526d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.941+0000 7f526d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:34.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.941+0000 7f526d59a640 10 monclient: dump: 2024-03-25T19:30:34.941 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:34.941 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:34.941 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:34.941 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:34.941 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.941 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:34.941 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:34.941 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:34.941 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:34.941 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:34.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.942+0000 7f5274a8d640 10 monclient: shutdown 2024-03-25T19:30:34.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.942+0000 7f5274a8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:34.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.942+0000 7f5274a8d640 1 -- 172.21.15.19:0/238228004 >> v1:172.21.15.188:6789/0 conn(0x7f527013fc60 legacy=0x7f527006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.942+0000 7f5274a8d640 1 -- 172.21.15.19:0/238228004 shutdown_connections 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.942+0000 7f5274a8d640 1 -- 172.21.15.19:0/238228004 >> 172.21.15.19:0/238228004 conn(0x7f52700b4a80 msgr2=0x7f52700b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.942+0000 7f5274a8d640 1 -- 172.21.15.19:0/238228004 shutdown_connections 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.942+0000 7f5274a8d640 1 -- 172.21.15.19:0/238228004 wait complete. 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.943+0000 7f5274a8d640 10 monclient: build_initial_monmap 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.943+0000 7f5274a8d640 10 monclient: monmap: 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:34.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.943+0000 7f5274a8d640 1 librados: starting msgr at 2024-03-25T19:30:34.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.943+0000 7f5274a8d640 1 librados: starting objecter 2024-03-25T19:30:34.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.943+0000 7f5274a8d640 1 Processor -- start 2024-03-25T19:30:34.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.943+0000 7f5274a8d640 1 -- start start 2024-03-25T19:30:34.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.943+0000 7f5274a8d640 1 librados: setting wanted keys 2024-03-25T19:30:34.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.943+0000 7f5274a8d640 1 librados: calling monclient init 2024-03-25T19:30:34.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.943+0000 7f5274a8d640 10 monclient: init 2024-03-25T19:30:34.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.944+0000 7f5274a8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:34.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.944+0000 7f5274a8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:34.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.944+0000 7f5274a8d640 10 monclient(hunting): picked mon.noname-c con 0x7f5270070570 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:34.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.944+0000 7f5274a8d640 10 monclient(hunting): picked mon.noname-b con 0x7f52700727a0 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:34.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.944+0000 7f5274a8d640 10 monclient(hunting): picked mon.noname-a con 0x7f527013fc60 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:34.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.944+0000 7f5274a8d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5270138cd0 con 0x7f527013fc60 2024-03-25T19:30:34.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.944+0000 7f5274a8d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527013a6d0 con 0x7f52700727a0 2024-03-25T19:30:34.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.944+0000 7f5274a8d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5270139b30 con 0x7f5270070570 2024-03-25T19:30:34.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.944+0000 7f5274a8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:34.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.944+0000 7f5274a8d640 10 monclient(hunting): authenticate will time out at 6088.213374s 2024-03-25T19:30:34.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.957+0000 7f526dd9b640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f52700727a0 0x7f5270139420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:35344/0 (socket says 172.21.15.19:35344) 2024-03-25T19:30:34.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.957+0000 7f526dd9b640 1 -- 172.21.15.19:0/444720290 learned_addr learned my addr 172.21.15.19:0/444720290 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:34.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.959+0000 7f525effd640 10 client.?.objecter ms_handle_connect 0x7f5270070570 2024-03-25T19:30:34.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.959+0000 7f525effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:34.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.961+0000 7f525effd640 10 client.?.objecter ms_handle_connect 0x7f52700727a0 2024-03-25T19:30:34.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.961+0000 7f525effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.967+0000 7f525effd640 10 client.?.objecter ms_handle_connect 0x7f527013fc60 2024-03-25T19:30:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.967+0000 7f525effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:34.979 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.980+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149803362 0 0) 0x7f5270139b30 con 0x7f5270070570 2024-03-25T19:30:34.979 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.980+0000 7f525effd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.979 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.980+0000 7f525effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.979 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.980+0000 7f525effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.979 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.980+0000 7f525effd640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:30:34.979 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.980+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5250004eb0 con 0x7f5270070570 2024-03-25T19:30:34.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.984+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693509820 0 0) 0x7f5270138cd0 con 0x7f527013fc60 2024-03-25T19:30:34.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.984+0000 7f525effd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.984+0000 7f525effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.984+0000 7f525effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.984+0000 7f525effd640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:30:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.984+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5270139b30 con 0x7f527013fc60 2024-03-25T19:30:34.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.991+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616887291 0 0) 0x7f527013a6d0 con 0x7f52700727a0 2024-03-25T19:30:34.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.991+0000 7f525effd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.991+0000 7f525effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.991+0000 7f525effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.991+0000 7f525effd640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:30:34.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:34.991+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5270138cd0 con 0x7f52700727a0 2024-03-25T19:30:35.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 940675168 0 0) 0x7f5270138cd0 con 0x7f52700727a0 2024-03-25T19:30:35.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f527013a6d0 con 0x7f52700727a0 2024-03-25T19:30:35.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5264001950 con 0x7f52700727a0 2024-03-25T19:30:35.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:35.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:35.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 10 monclient(hunting): dump: 2024-03-25T19:30:35.002 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4101924007 0 0) 0x7f5250004eb0 con 0x7f5270070570 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5270138cd0 con 0x7f5270070570 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5260001ba0 con 0x7f5270070570 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:35.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.003+0000 7f525effd640 10 monclient(hunting): dump: 2024-03-25T19:30:35.004 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:35.004 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:35.004 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:35.004 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:35.004 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:35.004 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:35.004 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:35.004 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:35.004 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:35.004 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:35.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.021+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 644610215 0 0) 0x7f527013a6d0 con 0x7f52700727a0 2024-03-25T19:30:35.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 >> v1:172.21.15.19:6790/0 conn(0x7f5270070570 legacy=0x7f52701385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:35.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 >> v1:172.21.15.19:6789/0 conn(0x7f527013fc60 legacy=0x7f5270132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:35.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f525effd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:35.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f525effd640 1 monclient: found mon.b 2024-03-25T19:30:35.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f525effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:35.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f525effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:35.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52701da9d0 con 0x7f52700727a0 2024-03-25T19:30:35.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f525effd640 10 monclient: _finish_auth 0 2024-03-25T19:30:35.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f525effd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:35.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f525effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:35.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f5274a8d640 5 monclient: authenticate success, global_id 4294 2024-03-25T19:30:35.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f5274a8d640 10 monclient: _renew_subs 2024-03-25T19:30:35.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f5274a8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:35.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f5274a8d640 1 -- 172.21.15.19:0/444720290 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52701dac80 con 0x7f52700727a0 2024-03-25T19:30:35.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f5274a8d640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:35.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f5274a8d640 10 monclient: _renew_subs 2024-03-25T19:30:35.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f5274a8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:35.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f5274a8d640 1 -- 172.21.15.19:0/444720290 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5270133600 con 0x7f52700727a0 2024-03-25T19:30:35.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.022+0000 7f5274a8d640 1 librados: init done 2024-03-25T19:30:35.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.023+0000 7f5274a8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:35.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.023+0000 7f5274a8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:35.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.023+0000 7f5274a8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:35.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.023+0000 7f5274a8d640 1 -- 172.21.15.19:0/444720290 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5270133900 con 0x7f52700727a0 2024-03-25T19:30:35.317 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.318+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52640047e0 con 0x7f52700727a0 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.318+0000 7f525effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.318+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5264001d80 con 0x7f52700727a0 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.318+0000 7f525effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.318+0000 7f525effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.318+0000 7f525effd640 10 monclient: dump: 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:35.318 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:35.319 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:35.319 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:35.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.318+0000 7f526cd99640 4 set_mon_vals no callback set 2024-03-25T19:30:35.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.319+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f526401a9a0 con 0x7f52700727a0 2024-03-25T19:30:35.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.320+0000 7f526e59c640 10 monclient: get_auth_request con 0x7f525006df20 auth_method 0 2024-03-25T19:30:35.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.320+0000 7f525effd640 10 client.4294.objecter ms_handle_connect 0x7f525006df20 2024-03-25T19:30:35.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.324+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f5264054780 con 0x7f52700727a0 2024-03-25T19:30:35.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.324+0000 7f525effd640 10 client.4294.objecter ms_dispatch 0x7f5270000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:35.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.324+0000 7f525effd640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:35.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.324+0000 7f525effd640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:35.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.324+0000 7f525effd640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T19:30:35.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.324+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52640500c0 con 0x7f52700727a0 2024-03-25T19:30:35.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.324+0000 7f525effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:35.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.324+0000 7f525effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:35.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.595+0000 7f5274a8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:35.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.595+0000 7f5274a8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:35.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.595+0000 7f5274a8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:35.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.596+0000 7f5274a8d640 1 -- 172.21.15.19:0/444720290 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5270139060 con 0x7f52700727a0 2024-03-25T19:30:35.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.622+0000 7f525effd640 1 -- 172.21.15.19:0/444720290 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f52640521c0 con 0x7f52700727a0 2024-03-25T19:30:35.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.622+0000 7f525effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:35.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.622+0000 7f525effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:35.621 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:30:35.621 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:35.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 10 librados: watch_flush enter 2024-03-25T19:30:35.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 10 librados: watch_flush exit 2024-03-25T19:30:35.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T19:30:35.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 10 client.4294.objecter successfully canceled tick 2024-03-25T19:30:35.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 1 -- 172.21.15.19:0/444720290 >> v1:172.21.15.188:6800/2207506229 conn(0x7f525006df20 legacy=0x7f52500703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:35.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 10 monclient: shutdown 2024-03-25T19:30:35.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:35.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 1 -- 172.21.15.19:0/444720290 >> v1:172.21.15.188:6789/0 conn(0x7f52700727a0 legacy=0x7f5270139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:35.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 1 -- 172.21.15.19:0/444720290 shutdown_connections 2024-03-25T19:30:35.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 1 -- 172.21.15.19:0/444720290 >> 172.21.15.19:0/444720290 conn(0x7f52700b4a80 msgr2=0x7f52700afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:35.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 1 -- 172.21.15.19:0/444720290 shutdown_connections 2024-03-25T19:30:35.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 1 -- 172.21.15.19:0/444720290 wait complete. 2024-03-25T19:30:35.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:35.625+0000 7f5274a8d640 1 librados: shutdown 2024-03-25T19:30:35.636 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:42.442 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:42.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.562+0000 7fc0c816d640 10 monclient: get_monmap_and_config 2024-03-25T19:30:42.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.562+0000 7fc0c816d640 10 monclient: build_initial_monmap 2024-03-25T19:30:42.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.562+0000 7fc0c816d640 10 monclient: monmap: 2024-03-25T19:30:42.562 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:42.563 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:42.563 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:30:42.563971+0000 2024-03-25T19:30:42.563 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:30:42.563971+0000 2024-03-25T19:30:42.563 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:42.563 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:42.563 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:42.563 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:30:42.563 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:30:42.563 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:42.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.563+0000 7fc0c816d640 1 Processor -- start 2024-03-25T19:30:42.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.563+0000 7fc0c816d640 1 -- start start 2024-03-25T19:30:42.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.563+0000 7fc0c816d640 10 monclient: init 2024-03-25T19:30:42.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.564+0000 7fc0c816d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:42.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.564+0000 7fc0c816d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:42.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.564+0000 7fc0c816d640 10 monclient(hunting): picked mon.noname-c con 0x7fc0c00ae6e0 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:42.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.564+0000 7fc0c816d640 10 monclient(hunting): picked mon.noname-b con 0x7fc0c006f080 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:42.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.564+0000 7fc0c816d640 10 monclient(hunting): picked mon.noname-a con 0x7fc0c00704e0 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:42.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.564+0000 7fc0c816d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c00718d0 con 0x7fc0c00704e0 2024-03-25T19:30:42.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.564+0000 7fc0c816d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c006fa30 con 0x7fc0c00ae6e0 2024-03-25T19:30:42.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.564+0000 7fc0c816d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c0070090 con 0x7fc0c006f080 2024-03-25T19:30:42.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.564+0000 7fc0c816d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:42.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.564+0000 7fc0c816d640 10 monclient(hunting): authenticate will time out at 6095.834180s 2024-03-25T19:30:42.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.568+0000 7fc0c670a640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fc0c00704e0 0x7fc0c014a910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:33062/0 (socket says 172.21.15.19:33062) 2024-03-25T19:30:42.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.568+0000 7fc0c670a640 1 -- 172.21.15.19:0/1660712066 learned_addr learned my addr 172.21.15.19:0/1660712066 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:42.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.578+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222184497 0 0) 0x7fc0c00718d0 con 0x7fc0c00704e0 2024-03-25T19:30:42.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.578+0000 7fc0c4f07640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.578+0000 7fc0c4f07640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.578+0000 7fc0c4f07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.578+0000 7fc0c4f07640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:30:42.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.578+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0ac005730 con 0x7fc0c00704e0 2024-03-25T19:30:42.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.581+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145735267 0 0) 0x7fc0c006fa30 con 0x7fc0c00ae6e0 2024-03-25T19:30:42.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.581+0000 7fc0c4f07640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.581+0000 7fc0c4f07640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.581+0000 7fc0c4f07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.581+0000 7fc0c4f07640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:30:42.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.581+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c00718d0 con 0x7fc0c00ae6e0 2024-03-25T19:30:42.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.588+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849732532 0 0) 0x7fc0c0070090 con 0x7fc0c006f080 2024-03-25T19:30:42.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.588+0000 7fc0c4f07640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.588+0000 7fc0c4f07640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.588+0000 7fc0c4f07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.588+0000 7fc0c4f07640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:30:42.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.588+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c006fa30 con 0x7fc0c006f080 2024-03-25T19:30:42.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3055570933 0 0) 0x7fc0ac005730 con 0x7fc0c00704e0 2024-03-25T19:30:42.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 >> v1:172.21.15.19:6790/0 conn(0x7fc0c006f080 legacy=0x7fc0c0071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 >> v1:172.21.15.188:6789/0 conn(0x7fc0c00ae6e0 legacy=0x7fc0c013c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 10 monclient: _finish_hunting 0 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 1 monclient: found mon.noname-a 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c006f510 con 0x7fc0c00704e0 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 10 monclient: _finish_auth 0 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 10 monclient: _check_auth_tickets 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc0b8001a10 con 0x7fc0c00704e0 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c4f07640 10 monclient: dump: 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:42.592 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:42.593 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:42.593 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:42.593 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.593 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:42.593 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:42.593 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:42.593 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:42.593 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:42.593 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c816d640 5 monclient: authenticate success, global_id 4302 2024-03-25T19:30:42.593 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.591+0000 7fc0c816d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:42.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.602+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0b8001b70 con 0x7fc0c00704e0 2024-03-25T19:30:42.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.602+0000 7fc0c4f07640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:42.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.602+0000 7fc0c4f07640 1 -- 172.21.15.19:0/1660712066 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc0b8001f80 con 0x7fc0c00704e0 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.602+0000 7fc0c816d640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.602+0000 7fc0c816d640 4 set_mon_vals no callback set 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.602+0000 7fc0c4f07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.602+0000 7fc0c4f07640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.602+0000 7fc0c4f07640 10 monclient: dump: 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:42.603 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:42.604 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:42.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.603+0000 7fc0c816d640 10 monclient: shutdown 2024-03-25T19:30:42.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.603+0000 7fc0c816d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:42.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.603+0000 7fc0c816d640 1 -- 172.21.15.19:0/1660712066 >> v1:172.21.15.19:6789/0 conn(0x7fc0c00704e0 legacy=0x7fc0c014a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.603+0000 7fc0c816d640 1 -- 172.21.15.19:0/1660712066 shutdown_connections 2024-03-25T19:30:42.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.603+0000 7fc0c816d640 1 -- 172.21.15.19:0/1660712066 >> 172.21.15.19:0/1660712066 conn(0x7fc0c006b630 msgr2=0x7fc0c00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:42.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.603+0000 7fc0c816d640 1 -- 172.21.15.19:0/1660712066 shutdown_connections 2024-03-25T19:30:42.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.603+0000 7fc0c816d640 1 -- 172.21.15.19:0/1660712066 wait complete. 2024-03-25T19:30:42.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.604+0000 7fc0c816d640 10 monclient: build_initial_monmap 2024-03-25T19:30:42.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.604+0000 7fc0c816d640 10 monclient: monmap: 2024-03-25T19:30:42.606 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:42.606 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:42.606 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.604+0000 7fc0c816d640 1 librados: starting msgr at 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.604+0000 7fc0c816d640 1 librados: starting objecter 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 1 Processor -- start 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 1 -- start start 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 1 librados: setting wanted keys 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 1 librados: calling monclient init 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 10 monclient: init 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:42.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 10 monclient(hunting): picked mon.noname-b con 0x7fc0c00704e0 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:42.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 10 monclient(hunting): picked mon.noname-c con 0x7fc0c00ae6e0 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:42.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 10 monclient(hunting): picked mon.noname-a con 0x7fc0c0134900 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:42.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c0141fc0 con 0x7fc0c0134900 2024-03-25T19:30:42.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c0142e50 con 0x7fc0c00704e0 2024-03-25T19:30:42.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c01dacb0 con 0x7fc0c00ae6e0 2024-03-25T19:30:42.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:42.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.605+0000 7fc0c816d640 10 monclient(hunting): authenticate will time out at 6095.875835s 2024-03-25T19:30:42.611 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.611+0000 7fc0c5f09640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fc0c00704e0 0x7fc0c013dfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:34664/0 (socket says 172.21.15.19:34664) 2024-03-25T19:30:42.611 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.611+0000 7fc0c5f09640 1 -- 172.21.15.19:0/2559582844 learned_addr learned my addr 172.21.15.19:0/2559582844 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:42.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.612+0000 7fc0b67fc640 10 client.?.objecter ms_handle_connect 0x7fc0c00704e0 2024-03-25T19:30:42.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.612+0000 7fc0b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:42.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.620+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315547443 0 0) 0x7fc0c0142e50 con 0x7fc0c00704e0 2024-03-25T19:30:42.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.620+0000 7fc0b67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.620+0000 7fc0b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.620+0000 7fc0b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.620+0000 7fc0b67fc640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:30:42.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.620+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a4004eb0 con 0x7fc0c00704e0 2024-03-25T19:30:42.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.621+0000 7fc0b67fc640 10 client.?.objecter ms_handle_connect 0x7fc0c0134900 2024-03-25T19:30:42.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.621+0000 7fc0b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:42.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.636+0000 7fc0b67fc640 10 client.?.objecter ms_handle_connect 0x7fc0c00ae6e0 2024-03-25T19:30:42.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.636+0000 7fc0b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:42.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.637+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751842968 0 0) 0x7fc0c0141fc0 con 0x7fc0c0134900 2024-03-25T19:30:42.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.637+0000 7fc0b67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.637+0000 7fc0b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.637+0000 7fc0b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.637+0000 7fc0b67fc640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:30:42.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.637+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c0142e50 con 0x7fc0c0134900 2024-03-25T19:30:42.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.638+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 20497628 0 0) 0x7fc0a4004eb0 con 0x7fc0c00704e0 2024-03-25T19:30:42.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.638+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0c0141fc0 con 0x7fc0c00704e0 2024-03-25T19:30:42.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.638+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc0bc001990 con 0x7fc0c00704e0 2024-03-25T19:30:42.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.638+0000 7fc0b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.638+0000 7fc0b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:42.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.638+0000 7fc0b67fc640 10 monclient(hunting): dump: 2024-03-25T19:30:42.639 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:42.639 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:42.639 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:42.639 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:42.639 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.639 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:42.639 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:42.639 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:42.639 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:42.639 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:42.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.638+0000 7fc0b67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:42.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.649+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103798920 0 0) 0x7fc0c01dacb0 con 0x7fc0c00ae6e0 2024-03-25T19:30:42.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.649+0000 7fc0b67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.649+0000 7fc0b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.649+0000 7fc0b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.649+0000 7fc0b67fc640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:30:42.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.649+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a4004eb0 con 0x7fc0c00ae6e0 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.654+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 747394770 0 0) 0x7fc0c0142e50 con 0x7fc0c0134900 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.654+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0c01dacb0 con 0x7fc0c0134900 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.654+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc0b8001e80 con 0x7fc0c0134900 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.654+0000 7fc0b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.654+0000 7fc0b67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.654+0000 7fc0b67fc640 10 monclient(hunting): dump: 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:42.654 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:42.655 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:42.655 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:42.655 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:42.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1002111355 0 0) 0x7fc0c0141fc0 con 0x7fc0c00704e0 2024-03-25T19:30:42.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 >> v1:172.21.15.19:6790/0 conn(0x7fc0c00ae6e0 legacy=0x7fc0c0140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 >> v1:172.21.15.19:6789/0 conn(0x7fc0c0134900 legacy=0x7fc0c01411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0b67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:42.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0b67fc640 1 monclient: found mon.b 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0b67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c01dae40 con 0x7fc0c00704e0 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0b67fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0b67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0c816d640 5 monclient: authenticate success, global_id 4300 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0c816d640 10 monclient: _renew_subs 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0c816d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0c816d640 1 -- 172.21.15.19:0/2559582844 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c01418f0 con 0x7fc0c00704e0 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0c816d640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0c816d640 10 monclient: _renew_subs 2024-03-25T19:30:42.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0c816d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:42.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0c816d640 1 -- 172.21.15.19:0/2559582844 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c01db430 con 0x7fc0c00704e0 2024-03-25T19:30:42.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.663+0000 7fc0c816d640 1 librados: init done 2024-03-25T19:30:42.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.664+0000 7fc0c816d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:42.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.664+0000 7fc0c816d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:42.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.664+0000 7fc0c816d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:42.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.664+0000 7fc0c816d640 1 -- 172.21.15.19:0/2559582844 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c00627b0 con 0x7fc0c00704e0 2024-03-25T19:30:42.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.941+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0bc0035f0 con 0x7fc0c00704e0 2024-03-25T19:30:42.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.941+0000 7fc0b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:42.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.941+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc0bc003980 con 0x7fc0c00704e0 2024-03-25T19:30:42.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.941+0000 7fc0b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.941+0000 7fc0b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:42.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.941+0000 7fc0b67fc640 10 monclient: dump: 2024-03-25T19:30:42.942 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:42.942 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:42.942 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:42.942 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:42.942 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.942 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:42.942 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:42.942 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:42.942 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:42.942 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:42.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.941+0000 7fc0b7fff640 4 set_mon_vals no callback set 2024-03-25T19:30:42.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.942+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fc0bc003ba0 con 0x7fc0c00704e0 2024-03-25T19:30:42.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.943+0000 7fc0c5708640 10 monclient: get_auth_request con 0x7fc0a406dfb0 auth_method 0 2024-03-25T19:30:42.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.943+0000 7fc0b67fc640 10 client.4300.objecter ms_handle_connect 0x7fc0a406dfb0 2024-03-25T19:30:42.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.948+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fc0bc054790 con 0x7fc0c00704e0 2024-03-25T19:30:42.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.948+0000 7fc0b67fc640 10 client.4300.objecter ms_dispatch 0x7fc0c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:42.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.948+0000 7fc0b67fc640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:42.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.948+0000 7fc0b67fc640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:42.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.948+0000 7fc0b67fc640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-25T19:30:42.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.949+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0bc0502a0 con 0x7fc0c00704e0 2024-03-25T19:30:42.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.949+0000 7fc0b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:42.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:42.949+0000 7fc0b67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:43.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.173+0000 7fc0c816d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:43.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.173+0000 7fc0c816d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:43.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.173+0000 7fc0c816d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:43.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.173+0000 7fc0c816d640 1 -- 172.21.15.19:0/2559582844 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0c0074d80 con 0x7fc0c00704e0 2024-03-25T19:30:43.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.196+0000 7fc0b67fc640 1 -- 172.21.15.19:0/2559582844 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fc0c0074d80 con 0x7fc0c00704e0 2024-03-25T19:30:43.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.196+0000 7fc0b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:43.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.196+0000 7fc0b67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:43.197 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:30:43.197 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:43.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 10 librados: watch_flush enter 2024-03-25T19:30:43.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 10 librados: watch_flush exit 2024-03-25T19:30:43.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-25T19:30:43.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 10 client.4300.objecter successfully canceled tick 2024-03-25T19:30:43.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 1 -- 172.21.15.19:0/2559582844 >> v1:172.21.15.188:6800/2207506229 conn(0x7fc0a406dfb0 legacy=0x7fc0a4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:43.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 10 monclient: shutdown 2024-03-25T19:30:43.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:43.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 1 -- 172.21.15.19:0/2559582844 >> v1:172.21.15.188:6789/0 conn(0x7fc0c00704e0 legacy=0x7fc0c013dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:43.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 1 -- 172.21.15.19:0/2559582844 shutdown_connections 2024-03-25T19:30:43.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 1 -- 172.21.15.19:0/2559582844 >> 172.21.15.19:0/2559582844 conn(0x7fc0c006b630 msgr2=0x7fc0c00b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:43.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 1 -- 172.21.15.19:0/2559582844 shutdown_connections 2024-03-25T19:30:43.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 1 -- 172.21.15.19:0/2559582844 wait complete. 2024-03-25T19:30:43.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:43.199+0000 7fc0c816d640 1 librados: shutdown 2024-03-25T19:30:43.211 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:50.018 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:50.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.131+0000 7f87bdc0d640 10 monclient: get_monmap_and_config 2024-03-25T19:30:50.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.131+0000 7f87bdc0d640 10 monclient: build_initial_monmap 2024-03-25T19:30:50.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.131+0000 7f87bdc0d640 10 monclient: monmap: 2024-03-25T19:30:50.131 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:50.131 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:50.131 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:30:50.132330+0000 2024-03-25T19:30:50.131 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:30:50.132330+0000 2024-03-25T19:30:50.131 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:50.131 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:50.131 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:50.131 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:30:50.131 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:30:50.131 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:50.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.132+0000 7f87bdc0d640 1 Processor -- start 2024-03-25T19:30:50.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.132+0000 7f87bdc0d640 1 -- start start 2024-03-25T19:30:50.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.132+0000 7f87bdc0d640 10 monclient: init 2024-03-25T19:30:50.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.132+0000 7f87bdc0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:50.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.132+0000 7f87bdc0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:50.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.132+0000 7f87bdc0d640 10 monclient(hunting): picked mon.noname-b con 0x7f87b80a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:50.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.133+0000 7f87bdc0d640 10 monclient(hunting): picked mon.noname-a con 0x7f87b813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:50.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.133+0000 7f87bdc0d640 10 monclient(hunting): picked mon.noname-c con 0x7f87b81430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:50.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.133+0000 7f87bdc0d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b8142520 con 0x7f87b813f710 2024-03-25T19:30:50.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.133+0000 7f87bdc0d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b80b28a0 con 0x7f87b81430c0 2024-03-25T19:30:50.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.133+0000 7f87bdc0d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b80b2a30 con 0x7f87b80a98f0 2024-03-25T19:30:50.134 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.133+0000 7f87bdc0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:50.134 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.133+0000 7f87bdc0d640 10 monclient(hunting): authenticate will time out at 6103.402999s 2024-03-25T19:30:50.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.140+0000 7f87b7fff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f87b81430c0 0x7f87b814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:57812/0 (socket says 172.21.15.19:57812) 2024-03-25T19:30:50.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.140+0000 7f87b7fff640 1 -- 172.21.15.19:0/1070590824 learned_addr learned my addr 172.21.15.19:0/1070590824 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:50.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.152+0000 7f87b67fc640 1 -- 172.21.15.19:0/1070590824 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762786624 0 0) 0x7f87b80b28a0 con 0x7f87b81430c0 2024-03-25T19:30:50.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.152+0000 7f87b67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:50.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.152+0000 7f87b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:50.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.152+0000 7f87b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:50.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.152+0000 7f87b67fc640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:30:50.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.152+0000 7f87b67fc640 1 -- 172.21.15.19:0/1070590824 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a0003520 con 0x7f87b81430c0 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.157+0000 7f87b67fc640 1 -- 172.21.15.19:0/1070590824 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748551075 0 0) 0x7f87b8142520 con 0x7f87b813f710 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.157+0000 7f87b67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.157+0000 7f87b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.157+0000 7f87b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.157+0000 7f87b67fc640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.157+0000 7f87b67fc640 1 -- 172.21.15.19:0/1070590824 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b80b28a0 con 0x7f87b813f710 2024-03-25T19:30:50.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 1 -- 172.21.15.19:0/1070590824 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2210141155 0 0) 0x7f87a0003520 con 0x7f87b81430c0 2024-03-25T19:30:50.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 1 -- 172.21.15.19:0/1070590824 >> v1:172.21.15.19:6790/0 conn(0x7f87b80a98f0 legacy=0x7f87b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 1 -- 172.21.15.19:0/1070590824 >> v1:172.21.15.19:6789/0 conn(0x7f87b813f710 legacy=0x7f87b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 1 monclient: found mon.noname-c 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 1 -- 172.21.15.19:0/1070590824 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b80b44c0 con 0x7f87b81430c0 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 1 -- 172.21.15.19:0/1070590824 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f87b0001a10 con 0x7f87b81430c0 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87bdc0d640 5 monclient: authenticate success, global_id 4303 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87bdc0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87b67fc640 10 monclient: dump: 2024-03-25T19:30:50.172 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:50.173 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:50.173 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:50.173 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:50.173 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.173 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:50.173 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:50.173 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:50.173 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:50.173 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:50.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.171+0000 7f87bdc0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:50.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.191+0000 7f87b67fc640 1 -- 172.21.15.19:0/1070590824 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87b0001b70 con 0x7f87b81430c0 2024-03-25T19:30:50.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.191+0000 7f87b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:50.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.191+0000 7f87b67fc640 1 -- 172.21.15.19:0/1070590824 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f87b0001f80 con 0x7f87b81430c0 2024-03-25T19:30:50.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.191+0000 7f87b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.191+0000 7f87b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:50.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.191+0000 7f87b67fc640 10 monclient: dump: 2024-03-25T19:30:50.191 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:50.191 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:50.191 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:50.191 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:50.192 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.192 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:50.192 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:50.192 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:50.192 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:50.192 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:50.192 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.191+0000 7f87bdc0d640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:50.192 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.191+0000 7f87bdc0d640 4 set_mon_vals no callback set 2024-03-25T19:30:50.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.192+0000 7f87bdc0d640 10 monclient: shutdown 2024-03-25T19:30:50.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.192+0000 7f87bdc0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.192+0000 7f87bdc0d640 1 -- 172.21.15.19:0/1070590824 >> v1:172.21.15.188:6789/0 conn(0x7f87b81430c0 legacy=0x7f87b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.192+0000 7f87bdc0d640 1 -- 172.21.15.19:0/1070590824 shutdown_connections 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.192+0000 7f87bdc0d640 1 -- 172.21.15.19:0/1070590824 >> 172.21.15.19:0/1070590824 conn(0x7f87b80b7ea0 msgr2=0x7f87b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.192+0000 7f87bdc0d640 1 -- 172.21.15.19:0/1070590824 shutdown_connections 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.192+0000 7f87bdc0d640 1 -- 172.21.15.19:0/1070590824 wait complete. 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.193+0000 7f87bdc0d640 10 monclient: build_initial_monmap 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.193+0000 7f87bdc0d640 10 monclient: monmap: 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:50.194 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:30:50.195 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:50.195 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:30:50.195 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:30:50.195 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:50.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.193+0000 7f87bdc0d640 1 librados: starting msgr at 2024-03-25T19:30:50.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.193+0000 7f87bdc0d640 1 librados: starting objecter 2024-03-25T19:30:50.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.194+0000 7f87bdc0d640 1 Processor -- start 2024-03-25T19:30:50.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.194+0000 7f87bdc0d640 1 -- start start 2024-03-25T19:30:50.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.194+0000 7f87bdc0d640 1 librados: setting wanted keys 2024-03-25T19:30:50.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.194+0000 7f87bdc0d640 1 librados: calling monclient init 2024-03-25T19:30:50.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.194+0000 7f87bdc0d640 10 monclient: init 2024-03-25T19:30:50.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.194+0000 7f87bdc0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:50.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.195+0000 7f87bdc0d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:50.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.195+0000 7f87bdc0d640 10 monclient(hunting): picked mon.noname-a con 0x7f87b80a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:50.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.195+0000 7f87bdc0d640 10 monclient(hunting): picked mon.noname-b con 0x7f87b813f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:50.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.195+0000 7f87bdc0d640 10 monclient(hunting): picked mon.noname-c con 0x7f87b81430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:50.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.195+0000 7f87bdc0d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b8135230 con 0x7f87b80a98f0 2024-03-25T19:30:50.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.195+0000 7f87bdc0d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b8135da0 con 0x7f87b813f710 2024-03-25T19:30:50.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.195+0000 7f87bdc0d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b8135f30 con 0x7f87b81430c0 2024-03-25T19:30:50.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.195+0000 7f87bdc0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:50.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.195+0000 7f87bdc0d640 10 monclient(hunting): authenticate will time out at 6103.465104s 2024-03-25T19:30:50.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.196+0000 7f87b7fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f87b81430c0 0x7f87b81db560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:37260/0 (socket says 172.21.15.19:37260) 2024-03-25T19:30:50.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.196+0000 7f87b7fff640 1 -- 172.21.15.19:0/4238991376 learned_addr learned my addr 172.21.15.19:0/4238991376 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:50.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.198+0000 7f879ffff640 10 client.?.objecter ms_handle_connect 0x7f87b81430c0 2024-03-25T19:30:50.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.198+0000 7f879ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:50.203 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.203+0000 7f879ffff640 10 client.?.objecter ms_handle_connect 0x7f87b813f710 2024-03-25T19:30:50.203 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.203+0000 7f879ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:50.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.206+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433874043 0 0) 0x7f87b8135da0 con 0x7f87b813f710 2024-03-25T19:30:50.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.206+0000 7f879ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:50.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.206+0000 7f879ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:50.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.206+0000 7f879ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:50.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.206+0000 7f879ffff640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:30:50.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.206+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a4004eb0 con 0x7f87b813f710 2024-03-25T19:30:50.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.209+0000 7f879ffff640 10 client.?.objecter ms_handle_connect 0x7f87b80a98f0 2024-03-25T19:30:50.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.209+0000 7f879ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:50.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.214+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053292324 0 0) 0x7f87b8135f30 con 0x7f87b81430c0 2024-03-25T19:30:50.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.214+0000 7f879ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:50.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.214+0000 7f879ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:50.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.214+0000 7f879ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:50.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.214+0000 7f879ffff640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:30:50.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.215+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b8135da0 con 0x7f87b81430c0 2024-03-25T19:30:50.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.221+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214651011 0 0) 0x7f87b8135230 con 0x7f87b80a98f0 2024-03-25T19:30:50.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.221+0000 7f879ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:50.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.221+0000 7f879ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:50.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.221+0000 7f879ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:50.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.221+0000 7f879ffff640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:30:50.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.221+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b8135f30 con 0x7f87b80a98f0 2024-03-25T19:30:50.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.229+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3571275765 0 0) 0x7f87a4004eb0 con 0x7f87b813f710 2024-03-25T19:30:50.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.229+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87b8135230 con 0x7f87b813f710 2024-03-25T19:30:50.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.229+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f87ac001930 con 0x7f87b813f710 2024-03-25T19:30:50.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.229+0000 7f879ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.229+0000 7f879ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:50.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.229+0000 7f879ffff640 10 monclient(hunting): dump: 2024-03-25T19:30:50.229 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:50.229 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:50.229 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:50.230 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:50.230 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.230 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:50.230 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:50.230 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:50.230 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:50.230 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:50.230 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.229+0000 7f879ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:50.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.231+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3582404382 0 0) 0x7f87b8135da0 con 0x7f87b81430c0 2024-03-25T19:30:50.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.231+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87a4004eb0 con 0x7f87b81430c0 2024-03-25T19:30:50.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.231+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f87b0001e80 con 0x7f87b81430c0 2024-03-25T19:30:50.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.231+0000 7f879ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.231+0000 7f879ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:50.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.231+0000 7f879ffff640 10 monclient(hunting): dump: 2024-03-25T19:30:50.232 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:50.232 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:50.232 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:50.232 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:50.232 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.232 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:50.232 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:50.232 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:50.232 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:50.232 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:50.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.240+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1935296510 0 0) 0x7f87b8135f30 con 0x7f87b80a98f0 2024-03-25T19:30:50.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.240+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87b8135da0 con 0x7f87b80a98f0 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.240+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f87a8001830 con 0x7f87b80a98f0 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.240+0000 7f879ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.240+0000 7f879ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.240+0000 7f879ffff640 10 monclient(hunting): dump: 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:50.241 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:50.242 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:50.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1813754035 0 0) 0x7f87b8135230 con 0x7f87b813f710 2024-03-25T19:30:50.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 >> v1:172.21.15.19:6790/0 conn(0x7f87b81430c0 legacy=0x7f87b81db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 >> v1:172.21.15.19:6789/0 conn(0x7f87b80a98f0 legacy=0x7f87b81d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f879ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f879ffff640 1 monclient: found mon.b 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f879ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f879ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b81360c0 con 0x7f87b813f710 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f879ffff640 10 monclient: _finish_auth 0 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f879ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f879ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f87bdc0d640 5 monclient: authenticate success, global_id 4306 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f87bdc0d640 10 monclient: _renew_subs 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f87bdc0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f87bdc0d640 1 -- 172.21.15.19:0/4238991376 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b81d5000 con 0x7f87b813f710 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f87bdc0d640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:50.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f87bdc0d640 10 monclient: _renew_subs 2024-03-25T19:30:50.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f87bdc0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:50.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f87bdc0d640 1 -- 172.21.15.19:0/4238991376 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b81d5350 con 0x7f87b813f710 2024-03-25T19:30:50.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.246+0000 7f87bdc0d640 1 librados: init done 2024-03-25T19:30:50.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.247+0000 7f87bdc0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:50.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.247+0000 7f87bdc0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:50.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.247+0000 7f87bdc0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:50.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.247+0000 7f87bdc0d640 1 -- 172.21.15.19:0/4238991376 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b81d4530 con 0x7f87b813f710 2024-03-25T19:30:50.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.522+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87ac003610 con 0x7f87b813f710 2024-03-25T19:30:50.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.522+0000 7f879ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:50.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.522+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f87ac003920 con 0x7f87b813f710 2024-03-25T19:30:50.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.522+0000 7f879ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.522+0000 7f879ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:50.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.522+0000 7f879ffff640 10 monclient: dump: 2024-03-25T19:30:50.523 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:50.523 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:50.523 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:50.523 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:50.524 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.524 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:50.524 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:50.524 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:50.524 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:50.524 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:50.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.522+0000 7f87b5ffb640 4 set_mon_vals no callback set 2024-03-25T19:30:50.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.523+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f87ac019930 con 0x7f87b813f710 2024-03-25T19:30:50.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.524+0000 7f87b77fe640 10 monclient: get_auth_request con 0x7f87a406e0b0 auth_method 0 2024-03-25T19:30:50.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.524+0000 7f879ffff640 10 client.4306.objecter ms_handle_connect 0x7f87a406e0b0 2024-03-25T19:30:50.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.529+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f87ac053620 con 0x7f87b813f710 2024-03-25T19:30:50.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.529+0000 7f879ffff640 10 client.4306.objecter ms_dispatch 0x7f87b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:50.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.529+0000 7f879ffff640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:50.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.529+0000 7f879ffff640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:50.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.529+0000 7f879ffff640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T19:30:50.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.529+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87ac04ef60 con 0x7f87b813f710 2024-03-25T19:30:50.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.529+0000 7f879ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:50.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.529+0000 7f879ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:50.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.786+0000 7f87bdc0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:50.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.786+0000 7f87bdc0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:50.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.786+0000 7f87bdc0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:30:50.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.786+0000 7f87bdc0d640 1 -- 172.21.15.19:0/4238991376 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87b8135090 con 0x7f87b813f710 2024-03-25T19:30:50.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.806+0000 7f879ffff640 1 -- 172.21.15.19:0/4238991376 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f87b8135090 con 0x7f87b813f710 2024-03-25T19:30:50.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.806+0000 7f879ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:50.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.806+0000 7f879ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:50.806 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:30:50.807 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:50.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.808+0000 7f87bdc0d640 10 librados: watch_flush enter 2024-03-25T19:30:50.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.808+0000 7f87bdc0d640 10 librados: watch_flush exit 2024-03-25T19:30:50.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.808+0000 7f87bdc0d640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T19:30:50.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.808+0000 7f87bdc0d640 10 client.4306.objecter successfully canceled tick 2024-03-25T19:30:50.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.808+0000 7f87bdc0d640 1 -- 172.21.15.19:0/4238991376 >> v1:172.21.15.188:6800/2207506229 conn(0x7f87a406e0b0 legacy=0x7f87a4070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.808+0000 7f87bdc0d640 10 monclient: shutdown 2024-03-25T19:30:50.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.808+0000 7f87bdc0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:50.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.808+0000 7f87bdc0d640 1 -- 172.21.15.19:0/4238991376 >> v1:172.21.15.188:6789/0 conn(0x7f87b813f710 legacy=0x7f87b81dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.808+0000 7f87bdc0d640 1 -- 172.21.15.19:0/4238991376 shutdown_connections 2024-03-25T19:30:50.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.808+0000 7f87bdc0d640 1 -- 172.21.15.19:0/4238991376 >> 172.21.15.19:0/4238991376 conn(0x7f87b80b7ea0 msgr2=0x7f87b814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:50.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.809+0000 7f87bdc0d640 1 -- 172.21.15.19:0/4238991376 shutdown_connections 2024-03-25T19:30:50.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.809+0000 7f87bdc0d640 1 -- 172.21.15.19:0/4238991376 wait complete. 2024-03-25T19:30:50.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:50.809+0000 7f87bdc0d640 1 librados: shutdown 2024-03-25T19:30:50.820 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:51.888 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:14.618 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:51.939 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:14.668 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:51.980 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:14.709 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:51.996 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:14.725 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:51.999 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:02:14.702 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:52.010 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:14.740 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:52.015 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:02:14.715 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:52.043 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:02:14.746 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:52.048 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:14.778 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:57.627 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:57.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.746+0000 7fba33b84640 10 monclient: get_monmap_and_config 2024-03-25T19:30:57.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.746+0000 7fba33b84640 10 monclient: build_initial_monmap 2024-03-25T19:30:57.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.746+0000 7fba33b84640 10 monclient: monmap: 2024-03-25T19:30:57.746 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:57.746 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:57.746 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:30:57.747621+0000 2024-03-25T19:30:57.746 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:30:57.747621+0000 2024-03-25T19:30:57.747 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:57.747 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:57.747 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:57.747 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:30:57.747 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:30:57.747 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:57.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.747+0000 7fba33b84640 1 Processor -- start 2024-03-25T19:30:57.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.747+0000 7fba33b84640 1 -- start start 2024-03-25T19:30:57.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.747+0000 7fba33b84640 10 monclient: init 2024-03-25T19:30:57.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.747+0000 7fba33b84640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:57.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.748+0000 7fba33b84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:57.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.748+0000 7fba33b84640 10 monclient(hunting): picked mon.noname-c con 0x7fba2c0a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:57.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.748+0000 7fba33b84640 10 monclient(hunting): picked mon.noname-b con 0x7fba2c13f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:57.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.748+0000 7fba33b84640 10 monclient(hunting): picked mon.noname-a con 0x7fba2c1430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:57.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.748+0000 7fba33b84640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2c142520 con 0x7fba2c1430c0 2024-03-25T19:30:57.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.748+0000 7fba33b84640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2c0b28a0 con 0x7fba2c0a98f0 2024-03-25T19:30:57.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.748+0000 7fba33b84640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2c0b2a30 con 0x7fba2c13f710 2024-03-25T19:30:57.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.748+0000 7fba33b84640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:57.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.748+0000 7fba33b84640 10 monclient(hunting): authenticate will time out at 6111.018043s 2024-03-25T19:30:57.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.757+0000 7fba32121640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fba2c1430c0 0x7fba2c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:52956/0 (socket says 172.21.15.19:52956) 2024-03-25T19:30:57.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.757+0000 7fba32121640 1 -- 172.21.15.19:0/106619984 learned_addr learned my addr 172.21.15.19:0/106619984 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:57.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.768+0000 7fba3091e640 1 -- 172.21.15.19:0/106619984 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610992855 0 0) 0x7fba2c142520 con 0x7fba2c1430c0 2024-03-25T19:30:57.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.768+0000 7fba3091e640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.768+0000 7fba3091e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.768+0000 7fba3091e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.768+0000 7fba3091e640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:30:57.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.768+0000 7fba3091e640 1 -- 172.21.15.19:0/106619984 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba18003520 con 0x7fba2c1430c0 2024-03-25T19:30:57.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.770+0000 7fba3091e640 1 -- 172.21.15.19:0/106619984 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125827570 0 0) 0x7fba2c0b28a0 con 0x7fba2c0a98f0 2024-03-25T19:30:57.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.771+0000 7fba3091e640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.771+0000 7fba3091e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.771+0000 7fba3091e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.771+0000 7fba3091e640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:30:57.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.771+0000 7fba3091e640 1 -- 172.21.15.19:0/106619984 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba2c142520 con 0x7fba2c0a98f0 2024-03-25T19:30:57.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.779+0000 7fba3091e640 1 -- 172.21.15.19:0/106619984 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1769607838 0 0) 0x7fba18003520 con 0x7fba2c1430c0 2024-03-25T19:30:57.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 1 -- 172.21.15.19:0/106619984 >> v1:172.21.15.19:6790/0 conn(0x7fba2c13f710 legacy=0x7fba2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 1 -- 172.21.15.19:0/106619984 >> v1:172.21.15.188:6789/0 conn(0x7fba2c0a98f0 legacy=0x7fba2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 10 monclient: _finish_hunting 0 2024-03-25T19:30:57.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 1 monclient: found mon.noname-a 2024-03-25T19:30:57.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:57.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:30:57.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 1 -- 172.21.15.19:0/106619984 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c0b44c0 con 0x7fba2c1430c0 2024-03-25T19:30:57.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 10 monclient: _finish_auth 0 2024-03-25T19:30:57.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 10 monclient: _check_auth_tickets 2024-03-25T19:30:57.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 1 -- 172.21.15.19:0/106619984 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fba24001a10 con 0x7fba2c1430c0 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba3091e640 10 monclient: dump: 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:57.781 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:57.782 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:57.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba33b84640 5 monclient: authenticate success, global_id 4314 2024-03-25T19:30:57.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.780+0000 7fba33b84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:57.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.795+0000 7fba3091e640 1 -- 172.21.15.19:0/106619984 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba24001b70 con 0x7fba2c1430c0 2024-03-25T19:30:57.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.795+0000 7fba3091e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:57.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.795+0000 7fba3091e640 1 -- 172.21.15.19:0/106619984 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fba24001f80 con 0x7fba2c1430c0 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.795+0000 7fba3091e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.795+0000 7fba3091e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.795+0000 7fba3091e640 10 monclient: dump: 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.795+0000 7fba33b84640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:57.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.795+0000 7fba33b84640 4 set_mon_vals no callback set 2024-03-25T19:30:57.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.796+0000 7fba33b84640 10 monclient: shutdown 2024-03-25T19:30:57.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.796+0000 7fba33b84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:57.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.796+0000 7fba33b84640 1 -- 172.21.15.19:0/106619984 >> v1:172.21.15.19:6789/0 conn(0x7fba2c1430c0 legacy=0x7fba2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.796+0000 7fba33b84640 1 -- 172.21.15.19:0/106619984 shutdown_connections 2024-03-25T19:30:57.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.796+0000 7fba33b84640 1 -- 172.21.15.19:0/106619984 >> 172.21.15.19:0/106619984 conn(0x7fba2c0b7ea0 msgr2=0x7fba2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:57.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.796+0000 7fba33b84640 1 -- 172.21.15.19:0/106619984 shutdown_connections 2024-03-25T19:30:57.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.796+0000 7fba33b84640 1 -- 172.21.15.19:0/106619984 wait complete. 2024-03-25T19:30:57.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.797+0000 7fba33b84640 10 monclient: build_initial_monmap 2024-03-25T19:30:57.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.797+0000 7fba33b84640 10 monclient: monmap: 2024-03-25T19:30:57.800 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:30:57.800 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:57.800 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:30:57.800 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:30:57.800 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:57.800 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:30:57.800 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:30:57.800 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:30:57.800 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.797+0000 7fba33b84640 1 librados: starting msgr at 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.797+0000 7fba33b84640 1 librados: starting objecter 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.797+0000 7fba33b84640 1 Processor -- start 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.797+0000 7fba33b84640 1 -- start start 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.797+0000 7fba33b84640 1 librados: setting wanted keys 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.797+0000 7fba33b84640 1 librados: calling monclient init 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.797+0000 7fba33b84640 10 monclient: init 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.798+0000 7fba33b84640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.798+0000 7fba33b84640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.798+0000 7fba33b84640 10 monclient(hunting): picked mon.noname-c con 0x7fba2c0a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.798+0000 7fba33b84640 10 monclient(hunting): picked mon.noname-a con 0x7fba2c13f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:30:57.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.798+0000 7fba33b84640 10 monclient(hunting): picked mon.noname-b con 0x7fba2c1430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:30:57.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.798+0000 7fba33b84640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2c139830 con 0x7fba2c13f710 2024-03-25T19:30:57.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.798+0000 7fba33b84640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2c1db7d0 con 0x7fba2c1430c0 2024-03-25T19:30:57.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.798+0000 7fba33b84640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2c1db410 con 0x7fba2c0a98f0 2024-03-25T19:30:57.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.798+0000 7fba33b84640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:57.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.798+0000 7fba33b84640 10 monclient(hunting): authenticate will time out at 6111.068495s 2024-03-25T19:30:57.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.807+0000 7fba31920640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fba2c0a98f0 0x7fba2c1dad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:41732/0 (socket says 172.21.15.19:41732) 2024-03-25T19:30:57.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.808+0000 7fba31920640 1 -- 172.21.15.19:0/1684217269 learned_addr learned my addr 172.21.15.19:0/1684217269 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:30:57.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.812+0000 7fba227fc640 10 client.?.objecter ms_handle_connect 0x7fba2c0a98f0 2024-03-25T19:30:57.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.812+0000 7fba227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:57.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.813+0000 7fba227fc640 10 client.?.objecter ms_handle_connect 0x7fba2c1430c0 2024-03-25T19:30:57.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.813+0000 7fba227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.820+0000 7fba227fc640 10 client.?.objecter ms_handle_connect 0x7fba2c13f710 2024-03-25T19:30:57.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.820+0000 7fba227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:57.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.825+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848991987 0 0) 0x7fba2c1db410 con 0x7fba2c0a98f0 2024-03-25T19:30:57.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.825+0000 7fba227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.825+0000 7fba227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.825+0000 7fba227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.825+0000 7fba227fc640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:30:57.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.825+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba10004eb0 con 0x7fba2c0a98f0 2024-03-25T19:30:57.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.826+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122180161 0 0) 0x7fba2c1db7d0 con 0x7fba2c1430c0 2024-03-25T19:30:57.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.826+0000 7fba227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.826+0000 7fba227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.826+0000 7fba227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.826+0000 7fba227fc640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:30:57.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.826+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba2c1db410 con 0x7fba2c1430c0 2024-03-25T19:30:57.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.829+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468497063 0 0) 0x7fba2c139830 con 0x7fba2c13f710 2024-03-25T19:30:57.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.829+0000 7fba227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.829+0000 7fba227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.829+0000 7fba227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.829+0000 7fba227fc640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:30:57.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.829+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba2c1db7d0 con 0x7fba2c13f710 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.838+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2616385496 0 0) 0x7fba2c1db7d0 con 0x7fba2c13f710 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.838+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba2c139830 con 0x7fba2c13f710 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.838+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fba1c001830 con 0x7fba2c13f710 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.838+0000 7fba227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.838+0000 7fba227fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.838+0000 7fba227fc640 10 monclient(hunting): dump: 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:57.839 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:57.839 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.839 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:57.839 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:57.839 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:57.839 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:57.839 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:57.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.838+0000 7fba227fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:57.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.842+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 851925025 0 0) 0x7fba10004eb0 con 0x7fba2c0a98f0 2024-03-25T19:30:57.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.842+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba2c1db7d0 con 0x7fba2c0a98f0 2024-03-25T19:30:57.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.842+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fba28001a10 con 0x7fba2c0a98f0 2024-03-25T19:30:57.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.842+0000 7fba227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.842+0000 7fba227fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:57.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.842+0000 7fba227fc640 10 monclient(hunting): dump: 2024-03-25T19:30:57.842 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:57.842 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:57.842 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:57.843 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:57.843 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.843 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:57.843 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:57.843 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:57.843 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:57.843 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.845+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2900039328 0 0) 0x7fba2c1db410 con 0x7fba2c1430c0 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.845+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba10004eb0 con 0x7fba2c1430c0 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.845+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fba24001e80 con 0x7fba2c1430c0 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.845+0000 7fba227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.845+0000 7fba227fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.845+0000 7fba227fc640 10 monclient(hunting): dump: 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:57.845 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:57.846 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:57.846 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:57.846 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:57.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3780091409 0 0) 0x7fba2c139830 con 0x7fba2c13f710 2024-03-25T19:30:57.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 >> v1:172.21.15.19:6790/0 conn(0x7fba2c0a98f0 legacy=0x7fba2c1dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 >> v1:172.21.15.188:6789/0 conn(0x7fba2c1430c0 legacy=0x7fba2c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba227fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:57.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba227fc640 1 monclient: found mon.a 2024-03-25T19:30:57.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:57.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba227fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:57.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c1db5a0 con 0x7fba2c13f710 2024-03-25T19:30:57.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba227fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:57.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba227fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:57.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba33b84640 5 monclient: authenticate success, global_id 4317 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba33b84640 10 monclient: _renew_subs 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba33b84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba33b84640 1 -- 172.21.15.19:0/1684217269 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2c0b1480 con 0x7fba2c13f710 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba33b84640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba33b84640 10 monclient: _renew_subs 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba33b84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba33b84640 1 -- 172.21.15.19:0/1684217269 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2c0b0b10 con 0x7fba2c13f710 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.850+0000 7fba33b84640 1 librados: init done 2024-03-25T19:30:57.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.852+0000 7fba33b84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:57.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.852+0000 7fba33b84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:57.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.852+0000 7fba33b84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:57.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:57.852+0000 7fba33b84640 1 -- 172.21.15.19:0/1684217269 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2c0b1780 con 0x7fba2c13f710 2024-03-25T19:30:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.172+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba1c001d40 con 0x7fba2c13f710 2024-03-25T19:30:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.172+0000 7fba227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.172+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fba1c003790 con 0x7fba2c13f710 2024-03-25T19:30:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.172+0000 7fba227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.172+0000 7fba227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.172+0000 7fba227fc640 10 monclient: dump: 2024-03-25T19:30:58.173 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:30:58.173 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:30:58.173 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:58.174 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:30:58.174 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:58.174 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:30:58.174 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:30:58.174 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:30:58.174 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:30:58.174 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:30:58.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.172+0000 7fba23fff640 4 set_mon_vals no callback set 2024-03-25T19:30:58.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.173+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fba1c0039b0 con 0x7fba2c13f710 2024-03-25T19:30:58.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.174+0000 7fba31920640 10 monclient: get_auth_request con 0x7fba1006e0b0 auth_method 0 2024-03-25T19:30:58.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.174+0000 7fba227fc640 10 client.4317.objecter ms_handle_connect 0x7fba1006e0b0 2024-03-25T19:30:58.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.180+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fba1c053590 con 0x7fba2c13f710 2024-03-25T19:30:58.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.180+0000 7fba227fc640 10 client.4317.objecter ms_dispatch 0x7fba2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:58.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.180+0000 7fba227fc640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:58.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.180+0000 7fba227fc640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:58.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.180+0000 7fba227fc640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-25T19:30:58.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.180+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba1c04eed0 con 0x7fba2c13f710 2024-03-25T19:30:58.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.180+0000 7fba227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:58.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.180+0000 7fba227fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:58.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.403+0000 7fba33b84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:58.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.403+0000 7fba33b84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:58.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.403+0000 7fba33b84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:30:58.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.403+0000 7fba33b84640 1 -- 172.21.15.19:0/1684217269 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba2c136510 con 0x7fba2c13f710 2024-03-25T19:30:58.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.441+0000 7fba227fc640 1 -- 172.21.15.19:0/1684217269 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fba2c136510 con 0x7fba2c13f710 2024-03-25T19:30:58.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.441+0000 7fba227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:58.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.441+0000 7fba227fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:58.441 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:30:58.441 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 10 librados: watch_flush enter 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 10 librados: watch_flush exit 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 10 client.4317.objecter successfully canceled tick 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 1 -- 172.21.15.19:0/1684217269 >> v1:172.21.15.188:6800/2207506229 conn(0x7fba1006e0b0 legacy=0x7fba10070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 10 monclient: shutdown 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 1 -- 172.21.15.19:0/1684217269 >> v1:172.21.15.19:6789/0 conn(0x7fba2c13f710 legacy=0x7fba2c0b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 1 -- 172.21.15.19:0/1684217269 shutdown_connections 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 1 -- 172.21.15.19:0/1684217269 >> 172.21.15.19:0/1684217269 conn(0x7fba2c0b7ea0 msgr2=0x7fba2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 1 -- 172.21.15.19:0/1684217269 shutdown_connections 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 1 -- 172.21.15.19:0/1684217269 wait complete. 2024-03-25T19:30:58.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:30:58.443+0000 7fba33b84640 1 librados: shutdown 2024-03-25T19:30:58.455 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:05.261 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:05.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.375+0000 7fa465a63640 10 monclient: get_monmap_and_config 2024-03-25T19:31:05.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.375+0000 7fa465a63640 10 monclient: build_initial_monmap 2024-03-25T19:31:05.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.375+0000 7fa465a63640 10 monclient: monmap: 2024-03-25T19:31:05.376 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:05.376 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:05.376 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:31:05.376899+0000 2024-03-25T19:31:05.376 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:31:05.376899+0000 2024-03-25T19:31:05.376 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:05.376 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:05.376 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:05.376 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:31:05.376 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:31:05.376 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:05.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.376+0000 7fa465a63640 1 Processor -- start 2024-03-25T19:31:05.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.376+0000 7fa465a63640 1 -- start start 2024-03-25T19:31:05.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.376+0000 7fa465a63640 10 monclient: init 2024-03-25T19:31:05.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.377+0000 7fa465a63640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:05.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.377+0000 7fa465a63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:05.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.377+0000 7fa465a63640 10 monclient(hunting): picked mon.noname-b con 0x7fa4600b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:05.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.377+0000 7fa465a63640 10 monclient(hunting): picked mon.noname-a con 0x7fa4600b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:05.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.377+0000 7fa465a63640 10 monclient(hunting): picked mon.noname-c con 0x7fa46014be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:05.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.377+0000 7fa465a63640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4600af940 con 0x7fa4600b1730 2024-03-25T19:31:05.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.377+0000 7fa465a63640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4600b0a70 con 0x7fa46014be90 2024-03-25T19:31:05.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.377+0000 7fa465a63640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4600b0d50 con 0x7fa4600b4d60 2024-03-25T19:31:05.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.377+0000 7fa465a63640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:05.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.377+0000 7fa465a63640 10 monclient(hunting): authenticate will time out at 6118.647330s 2024-03-25T19:31:05.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.383+0000 7fa45ffff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fa46014be90 0x7fa46014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:35682/0 (socket says 172.21.15.19:35682) 2024-03-25T19:31:05.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.383+0000 7fa45ffff640 1 -- 172.21.15.19:0/1067097726 learned_addr learned my addr 172.21.15.19:0/1067097726 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:05.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.402+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554665898 0 0) 0x7fa4600b0a70 con 0x7fa46014be90 2024-03-25T19:31:05.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.402+0000 7fa45e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:05.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.402+0000 7fa45e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:05.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.402+0000 7fa45e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:05.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.402+0000 7fa45e7fc640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:31:05.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.402+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa448003520 con 0x7fa46014be90 2024-03-25T19:31:05.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.403+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948026433 0 0) 0x7fa4600af940 con 0x7fa4600b1730 2024-03-25T19:31:05.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.403+0000 7fa45e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:05.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.403+0000 7fa45e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:05.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.403+0000 7fa45e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:05.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.403+0000 7fa45e7fc640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:31:05.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.403+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4600b0a70 con 0x7fa4600b1730 2024-03-25T19:31:05.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.406+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569631547 0 0) 0x7fa4600b0d50 con 0x7fa4600b4d60 2024-03-25T19:31:05.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.406+0000 7fa45e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:05.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.406+0000 7fa45e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:05.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.406+0000 7fa45e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:05.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.406+0000 7fa45e7fc640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:31:05.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.406+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4600af940 con 0x7fa4600b4d60 2024-03-25T19:31:05.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1688796880 0 0) 0x7fa4600b0a70 con 0x7fa4600b1730 2024-03-25T19:31:05.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 >> v1:172.21.15.19:6790/0 conn(0x7fa4600b4d60 legacy=0x7fa46013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 >> v1:172.21.15.188:6789/0 conn(0x7fa46014be90 legacy=0x7fa46014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:05.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 1 monclient: found mon.noname-a 2024-03-25T19:31:05.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:05.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:31:05.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4600b1030 con 0x7fa4600b1730 2024-03-25T19:31:05.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:05.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:05.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:05.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa454001670 con 0x7fa4600b1730 2024-03-25T19:31:05.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa45e7fc640 10 monclient: dump: 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa465a63640 5 monclient: authenticate success, global_id 4320 2024-03-25T19:31:05.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.412+0000 7fa465a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:05.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.423+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4540017d0 con 0x7fa4600b1730 2024-03-25T19:31:05.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.423+0000 7fa45e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:05.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.423+0000 7fa45e7fc640 1 -- 172.21.15.19:0/1067097726 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa454001be0 con 0x7fa4600b1730 2024-03-25T19:31:05.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.423+0000 7fa45e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.423+0000 7fa45e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.423+0000 7fa45e7fc640 10 monclient: dump: 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.423+0000 7fa465a63640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:05.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.423+0000 7fa465a63640 4 set_mon_vals no callback set 2024-03-25T19:31:05.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.424+0000 7fa465a63640 10 monclient: shutdown 2024-03-25T19:31:05.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.424+0000 7fa465a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:05.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.424+0000 7fa465a63640 1 -- 172.21.15.19:0/1067097726 >> v1:172.21.15.19:6789/0 conn(0x7fa4600b1730 legacy=0x7fa4600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.424+0000 7fa465a63640 1 -- 172.21.15.19:0/1067097726 shutdown_connections 2024-03-25T19:31:05.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.424+0000 7fa465a63640 1 -- 172.21.15.19:0/1067097726 >> 172.21.15.19:0/1067097726 conn(0x7fa4600617a0 msgr2=0x7fa460061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:05.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.424+0000 7fa465a63640 1 -- 172.21.15.19:0/1067097726 shutdown_connections 2024-03-25T19:31:05.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.424+0000 7fa465a63640 1 -- 172.21.15.19:0/1067097726 wait complete. 2024-03-25T19:31:05.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.425+0000 7fa465a63640 10 monclient: build_initial_monmap 2024-03-25T19:31:05.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.425+0000 7fa465a63640 10 monclient: monmap: 2024-03-25T19:31:05.426 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:05.426 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.426+0000 7fa465a63640 1 librados: starting msgr at 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.426+0000 7fa465a63640 1 librados: starting objecter 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.426+0000 7fa465a63640 1 Processor -- start 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.426+0000 7fa465a63640 1 -- start start 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.426+0000 7fa465a63640 1 librados: setting wanted keys 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.426+0000 7fa465a63640 1 librados: calling monclient init 2024-03-25T19:31:05.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.426+0000 7fa465a63640 10 monclient: init 2024-03-25T19:31:05.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.427+0000 7fa465a63640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:05.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.427+0000 7fa465a63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:05.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.427+0000 7fa465a63640 10 monclient(hunting): picked mon.noname-b con 0x7fa4600b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:05.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.427+0000 7fa465a63640 10 monclient(hunting): picked mon.noname-a con 0x7fa4600b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:05.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.427+0000 7fa465a63640 10 monclient(hunting): picked mon.noname-c con 0x7fa46014be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:05.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.427+0000 7fa465a63640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa460136940 con 0x7fa4600b4d60 2024-03-25T19:31:05.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.427+0000 7fa465a63640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4601d5050 con 0x7fa4600b1730 2024-03-25T19:31:05.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.427+0000 7fa465a63640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa460130570 con 0x7fa46014be90 2024-03-25T19:31:05.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.427+0000 7fa465a63640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:05.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.427+0000 7fa465a63640 10 monclient(hunting): authenticate will time out at 6118.697108s 2024-03-25T19:31:05.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.430+0000 7fa45f7fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fa4600b1730 0x7fa4601daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:35712/0 (socket says 172.21.15.19:35712) 2024-03-25T19:31:05.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.430+0000 7fa45f7fe640 1 -- 172.21.15.19:0/997062544 learned_addr learned my addr 172.21.15.19:0/997062544 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:05.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.431+0000 7fa447fff640 10 client.?.objecter ms_handle_connect 0x7fa4600b1730 2024-03-25T19:31:05.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.431+0000 7fa447fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:05.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.442+0000 7fa447fff640 10 client.?.objecter ms_handle_connect 0x7fa4600b4d60 2024-03-25T19:31:05.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.442+0000 7fa447fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:05.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.444+0000 7fa45effd640 1 -- 172.21.15.19:0/997062544 >> v1:172.21.15.19:6789/0 conn(0x7fa4600b4d60 legacy=0x7fa46012fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T19:31:05.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.444+0000 7fa45effd640 1 -- 172.21.15.19:0/997062544 >> v1:172.21.15.19:6789/0 conn(0x7fa4600b4d60 legacy=0x7fa46012fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:31:05.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.444+0000 7fa45effd640 1 --1- 172.21.15.19:0/997062544 >> v1:172.21.15.19:6789/0 conn(0x7fa4600b4d60 0x7fa46012fe60 :-1 s=OPENED pgs=11625 cs=1 l=1).handle_message read tag failed 2024-03-25T19:31:05.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.444+0000 7fa45effd640 1 --1- 172.21.15.19:0/997062544 >> v1:172.21.15.19:6789/0 conn(0x7fa4600b4d60 0x7fa46012fe60 :-1 s=OPENED pgs=11625 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:31:05.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.444+0000 7fa447fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.19:6789/0 2024-03-25T19:31:05.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.444+0000 7fa447fff640 10 client.?.objecter ms_handle_connect 0x7fa46014be90 2024-03-25T19:31:05.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.444+0000 7fa447fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:05.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.449+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720464285 0 0) 0x7fa4601d5050 con 0x7fa4600b1730 2024-03-25T19:31:05.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.449+0000 7fa447fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:05.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.449+0000 7fa447fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:05.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.449+0000 7fa447fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:05.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.449+0000 7fa447fff640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:31:05.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.449+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa44c004eb0 con 0x7fa4600b1730 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.460+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2507339823 0 0) 0x7fa44c004eb0 con 0x7fa4600b1730 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.460+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4601d5050 con 0x7fa4600b1730 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.460+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa450001a10 con 0x7fa4600b1730 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.460+0000 7fa447fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.460+0000 7fa447fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.460+0000 7fa447fff640 10 monclient(hunting): dump: 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.460+0000 7fa447fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:05.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.463+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969743192 0 0) 0x7fa460130570 con 0x7fa46014be90 2024-03-25T19:31:05.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.463+0000 7fa447fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:05.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.463+0000 7fa447fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:05.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.463+0000 7fa447fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:05.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.463+0000 7fa447fff640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:31:05.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.463+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa44c004eb0 con 0x7fa46014be90 2024-03-25T19:31:05.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.477+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2317389407 0 0) 0x7fa44c004eb0 con 0x7fa46014be90 2024-03-25T19:31:05.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa460130570 con 0x7fa46014be90 2024-03-25T19:31:05.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa458001a30 con 0x7fa46014be90 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 10 monclient(hunting): dump: 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1327256624 0 0) 0x7fa4601d5050 con 0x7fa4600b1730 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 >> v1:172.21.15.19:6790/0 conn(0x7fa46014be90 legacy=0x7fa4601e2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 >> v1:172.21.15.19:6789/0 conn(0x7fa4600b4d60 legacy=0x7fa46012fe60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 1 monclient: found mon.b 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa460130700 con 0x7fa4600b1730 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 10 monclient: _finish_auth 0 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa447fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa465a63640 5 monclient: authenticate success, global_id 4318 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa465a63640 10 monclient: _renew_subs 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa465a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa465a63640 1 -- 172.21.15.19:0/997062544 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4601380c0 con 0x7fa4600b1730 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa465a63640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa465a63640 10 monclient: _renew_subs 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa465a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa465a63640 1 -- 172.21.15.19:0/997062544 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4601457f0 con 0x7fa4600b1730 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.478+0000 7fa465a63640 1 librados: init done 2024-03-25T19:31:05.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.479+0000 7fa465a63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:05.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.479+0000 7fa465a63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:05.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.479+0000 7fa465a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:05.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.479+0000 7fa465a63640 1 -- 172.21.15.19:0/997062544 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa460136cb0 con 0x7fa4600b1730 2024-03-25T19:31:05.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.760+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa450001c70 con 0x7fa4600b1730 2024-03-25T19:31:05.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.760+0000 7fa447fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:05.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.760+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa450003a60 con 0x7fa4600b1730 2024-03-25T19:31:05.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.760+0000 7fa447fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:05.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.760+0000 7fa447fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:05.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.760+0000 7fa447fff640 10 monclient: dump: 2024-03-25T19:31:05.761 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:05.761 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:05.761 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:05.761 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:05.761 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:05.761 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:05.762 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:05.762 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:05.762 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:05.762 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:05.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.760+0000 7fa45dffb640 4 set_mon_vals no callback set 2024-03-25T19:31:05.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.761+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fa450003c40 con 0x7fa4600b1730 2024-03-25T19:31:05.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.762+0000 7fa45effd640 10 monclient: get_auth_request con 0x7fa44c06deb0 auth_method 0 2024-03-25T19:31:05.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.762+0000 7fa447fff640 10 client.4318.objecter ms_handle_connect 0x7fa44c06deb0 2024-03-25T19:31:05.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.766+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fa4500548c0 con 0x7fa4600b1730 2024-03-25T19:31:05.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.766+0000 7fa447fff640 10 client.4318.objecter ms_dispatch 0x7fa460000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:05.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.766+0000 7fa447fff640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:05.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.766+0000 7fa447fff640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:05.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.766+0000 7fa447fff640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-25T19:31:05.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.767+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa450050200 con 0x7fa4600b1730 2024-03-25T19:31:05.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.767+0000 7fa447fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:05.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:05.767+0000 7fa447fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:06.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.031+0000 7fa465a63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:06.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.031+0000 7fa465a63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:06.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.031+0000 7fa465a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:06.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.031+0000 7fa465a63640 1 -- 172.21.15.19:0/997062544 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4600627b0 con 0x7fa4600b1730 2024-03-25T19:31:06.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.083+0000 7fa447fff640 1 -- 172.21.15.19:0/997062544 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fa450052300 con 0x7fa4600b1730 2024-03-25T19:31:06.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.083+0000 7fa447fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:06.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.083+0000 7fa447fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:06.083 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:31:06.084 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:06.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.085+0000 7fa465a63640 10 librados: watch_flush enter 2024-03-25T19:31:06.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.085+0000 7fa465a63640 10 librados: watch_flush exit 2024-03-25T19:31:06.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.085+0000 7fa465a63640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-25T19:31:06.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.085+0000 7fa465a63640 10 client.4318.objecter successfully canceled tick 2024-03-25T19:31:06.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.085+0000 7fa465a63640 1 -- 172.21.15.19:0/997062544 >> v1:172.21.15.188:6800/2207506229 conn(0x7fa44c06deb0 legacy=0x7fa44c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:06.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.085+0000 7fa465a63640 10 monclient: shutdown 2024-03-25T19:31:06.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.085+0000 7fa465a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:06.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.085+0000 7fa465a63640 1 -- 172.21.15.19:0/997062544 >> v1:172.21.15.188:6789/0 conn(0x7fa4600b1730 legacy=0x7fa4601daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:06.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.085+0000 7fa465a63640 1 -- 172.21.15.19:0/997062544 shutdown_connections 2024-03-25T19:31:06.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.085+0000 7fa465a63640 1 -- 172.21.15.19:0/997062544 >> 172.21.15.19:0/997062544 conn(0x7fa4600617a0 msgr2=0x7fa4600b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:06.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.086+0000 7fa465a63640 1 -- 172.21.15.19:0/997062544 shutdown_connections 2024-03-25T19:31:06.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.086+0000 7fa465a63640 1 -- 172.21.15.19:0/997062544 wait complete. 2024-03-25T19:31:06.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:06.086+0000 7fa465a63640 1 librados: shutdown 2024-03-25T19:31:06.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:12.024 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:34.753 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.029 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:34.759 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.045 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:34.774 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.076 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:02:34.779 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.091 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:34.820 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.092 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:02:34.795 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.102 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:34.832 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.128 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:02:34.831 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.140 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:34.869 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.903 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:13.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.019+0000 7f61333ed640 10 monclient: get_monmap_and_config 2024-03-25T19:31:13.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.019+0000 7f61333ed640 10 monclient: build_initial_monmap 2024-03-25T19:31:13.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.019+0000 7f61333ed640 10 monclient: monmap: 2024-03-25T19:31:13.020 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:13.020 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:13.020 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:31:13.021136+0000 2024-03-25T19:31:13.020 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:31:13.021136+0000 2024-03-25T19:31:13.020 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:13.020 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:13.020 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:13.020 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:31:13.020 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:31:13.020 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:13.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.020+0000 7f61333ed640 1 Processor -- start 2024-03-25T19:31:13.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.020+0000 7f61333ed640 1 -- start start 2024-03-25T19:31:13.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.020+0000 7f61333ed640 10 monclient: init 2024-03-25T19:31:13.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.021+0000 7f61333ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:13.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.021+0000 7f61333ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.021+0000 7f61333ed640 10 monclient(hunting): picked mon.noname-b con 0x7f612c0a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.021+0000 7f61333ed640 10 monclient(hunting): picked mon.noname-c con 0x7f612c13f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.021+0000 7f61333ed640 10 monclient(hunting): picked mon.noname-a con 0x7f612c1430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.021+0000 7f61333ed640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c142520 con 0x7f612c1430c0 2024-03-25T19:31:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.021+0000 7f61333ed640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c0b28a0 con 0x7f612c13f710 2024-03-25T19:31:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.021+0000 7f61333ed640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c0b2a30 con 0x7f612c0a98f0 2024-03-25T19:31:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.021+0000 7f61333ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.021+0000 7f61333ed640 10 monclient(hunting): authenticate will time out at 6126.291632s 2024-03-25T19:31:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.022+0000 7f6130988640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f612c13f710 0x7f612c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:40406/0 (socket says 172.21.15.19:40406) 2024-03-25T19:31:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.022+0000 7f6130988640 1 -- 172.21.15.19:0/1076220003 learned_addr learned my addr 172.21.15.19:0/1076220003 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.121+0000 7f6123fff640 1 -- 172.21.15.19:0/1076220003 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329349226 0 0) 0x7f612c142520 con 0x7f612c1430c0 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.121+0000 7f6123fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.121+0000 7f6123fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.121+0000 7f6123fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.121+0000 7f6123fff640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.121+0000 7f6123fff640 1 -- 172.21.15.19:0/1076220003 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6118003520 con 0x7f612c1430c0 2024-03-25T19:31:13.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.143+0000 7f6123fff640 1 -- 172.21.15.19:0/1076220003 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038646523 0 0) 0x7f612c0b28a0 con 0x7f612c13f710 2024-03-25T19:31:13.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.143+0000 7f6123fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:13.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.143+0000 7f6123fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:13.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.143+0000 7f6123fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:13.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.143+0000 7f6123fff640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:31:13.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.143+0000 7f6123fff640 1 -- 172.21.15.19:0/1076220003 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f612c142520 con 0x7f612c13f710 2024-03-25T19:31:13.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 1 -- 172.21.15.19:0/1076220003 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2309165661 0 0) 0x7f6118003520 con 0x7f612c1430c0 2024-03-25T19:31:13.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 1 -- 172.21.15.19:0/1076220003 >> v1:172.21.15.19:6790/0 conn(0x7f612c0a98f0 legacy=0x7f612c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 1 -- 172.21.15.19:0/1076220003 >> v1:172.21.15.188:6789/0 conn(0x7f612c13f710 legacy=0x7f612c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 1 monclient: found mon.noname-a 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 1 -- 172.21.15.19:0/1076220003 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c0b44c0 con 0x7f612c1430c0 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 10 monclient: _finish_auth 0 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 1 -- 172.21.15.19:0/1076220003 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f6124001a10 con 0x7f612c1430c0 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f61333ed640 5 monclient: authenticate success, global_id 4326 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:13.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:13.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f6123fff640 10 monclient: dump: 2024-03-25T19:31:13.150 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:13.150 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:13.150 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:13.150 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:13.150 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:13.150 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:13.150 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:13.150 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:13.150 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:13.150 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:13.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.148+0000 7f61333ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:13.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.164+0000 7f6123fff640 1 -- 172.21.15.19:0/1076220003 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6124001b70 con 0x7f612c1430c0 2024-03-25T19:31:13.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.164+0000 7f6123fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:13.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.164+0000 7f6123fff640 1 -- 172.21.15.19:0/1076220003 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f6124001f80 con 0x7f612c1430c0 2024-03-25T19:31:13.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.164+0000 7f6123fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:13.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.164+0000 7f6123fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.164+0000 7f6123fff640 10 monclient: dump: 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.164+0000 7f61333ed640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:13.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.164+0000 7f61333ed640 4 set_mon_vals no callback set 2024-03-25T19:31:13.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.165+0000 7f61333ed640 10 monclient: shutdown 2024-03-25T19:31:13.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.165+0000 7f61333ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:13.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.165+0000 7f61333ed640 1 -- 172.21.15.19:0/1076220003 >> v1:172.21.15.19:6789/0 conn(0x7f612c1430c0 legacy=0x7f612c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.165+0000 7f61333ed640 1 -- 172.21.15.19:0/1076220003 shutdown_connections 2024-03-25T19:31:13.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.165+0000 7f61333ed640 1 -- 172.21.15.19:0/1076220003 >> 172.21.15.19:0/1076220003 conn(0x7f612c0b7ea0 msgr2=0x7f612c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:13.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.165+0000 7f61333ed640 1 -- 172.21.15.19:0/1076220003 shutdown_connections 2024-03-25T19:31:13.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.165+0000 7f61333ed640 1 -- 172.21.15.19:0/1076220003 wait complete. 2024-03-25T19:31:13.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.166+0000 7f61333ed640 10 monclient: build_initial_monmap 2024-03-25T19:31:13.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.166+0000 7f61333ed640 10 monclient: monmap: 2024-03-25T19:31:13.167 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 1 librados: starting msgr at 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 1 librados: starting objecter 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 1 Processor -- start 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 1 -- start start 2024-03-25T19:31:13.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 1 librados: setting wanted keys 2024-03-25T19:31:13.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 1 librados: calling monclient init 2024-03-25T19:31:13.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 10 monclient: init 2024-03-25T19:31:13.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:13.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:13.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 10 monclient(hunting): picked mon.noname-b con 0x7f612c0a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:13.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 10 monclient(hunting): picked mon.noname-a con 0x7f612c13f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:13.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 10 monclient(hunting): picked mon.noname-c con 0x7f612c1430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:13.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c139830 con 0x7f612c13f710 2024-03-25T19:31:13.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c1db7d0 con 0x7f612c0a98f0 2024-03-25T19:31:13.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612c1db410 con 0x7f612c1430c0 2024-03-25T19:31:13.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:13.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.167+0000 7f61333ed640 10 monclient(hunting): authenticate will time out at 6126.437720s 2024-03-25T19:31:13.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.168+0000 7f6131189640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f612c0a98f0 0x7f612c1dad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:40446/0 (socket says 172.21.15.19:40446) 2024-03-25T19:31:13.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.168+0000 7f6131189640 1 -- 172.21.15.19:0/3309444359 learned_addr learned my addr 172.21.15.19:0/3309444359 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:13.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.168+0000 7f6121ffb640 10 client.?.objecter ms_handle_connect 0x7f612c0a98f0 2024-03-25T19:31:13.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.168+0000 7f6121ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:13.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.186+0000 7f6121ffb640 10 client.?.objecter ms_handle_connect 0x7f612c13f710 2024-03-25T19:31:13.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.186+0000 7f6121ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:13.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.197+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765185894 0 0) 0x7f612c139830 con 0x7f612c13f710 2024-03-25T19:31:13.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.197+0000 7f6121ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:13.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.197+0000 7f6121ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:13.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.197+0000 7f6121ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:13.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.197+0000 7f6121ffb640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:31:13.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.197+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6110004eb0 con 0x7f612c13f710 2024-03-25T19:31:13.202 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.202+0000 7f6121ffb640 10 client.?.objecter ms_handle_connect 0x7f612c1430c0 2024-03-25T19:31:13.202 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.202+0000 7f6121ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:13.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.276+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411990518 0 0) 0x7f612c1db7d0 con 0x7f612c0a98f0 2024-03-25T19:31:13.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.276+0000 7f6121ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:13.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.276+0000 7f6121ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:13.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.276+0000 7f6121ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:13.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.276+0000 7f6121ffb640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:31:13.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.276+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f612c139830 con 0x7f612c0a98f0 2024-03-25T19:31:13.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.283+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3446021655 0 0) 0x7f6110004eb0 con 0x7f612c13f710 2024-03-25T19:31:13.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.283+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f612c1db7d0 con 0x7f612c13f710 2024-03-25T19:31:13.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.283+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f611c001930 con 0x7f612c13f710 2024-03-25T19:31:13.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.283+0000 7f6121ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:13.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.283+0000 7f6121ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:13.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.283+0000 7f6121ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:13.283 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:13.284 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:13.284 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:13.284 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:13.284 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:13.284 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:13.284 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:13.284 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:13.284 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:13.284 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:13.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.283+0000 7f6121ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2491051808 0 0) 0x7f612c1db7d0 con 0x7f612c13f710 2024-03-25T19:31:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 >> v1:172.21.15.19:6790/0 conn(0x7f612c1430c0 legacy=0x7f612c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 >> v1:172.21.15.188:6789/0 conn(0x7f612c0a98f0 legacy=0x7f612c1dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f6121ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f6121ffb640 1 monclient: found mon.a 2024-03-25T19:31:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f6121ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f6121ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612c1db5a0 con 0x7f612c13f710 2024-03-25T19:31:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f6121ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f6121ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f6121ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f61333ed640 5 monclient: authenticate success, global_id 4329 2024-03-25T19:31:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f61333ed640 10 monclient: _renew_subs 2024-03-25T19:31:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f61333ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f61333ed640 1 -- 172.21.15.19:0/3309444359 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c0b1480 con 0x7f612c13f710 2024-03-25T19:31:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f61333ed640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f61333ed640 10 monclient: _renew_subs 2024-03-25T19:31:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f61333ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f61333ed640 1 -- 172.21.15.19:0/3309444359 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c0b0b10 con 0x7f612c13f710 2024-03-25T19:31:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.296+0000 7f61333ed640 1 librados: init done 2024-03-25T19:31:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.297+0000 7f61333ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:13.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.297+0000 7f61333ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:13.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.297+0000 7f61333ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:13.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.297+0000 7f61333ed640 1 -- 172.21.15.19:0/3309444359 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c1db960 con 0x7f612c13f710 2024-03-25T19:31:13.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.634+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f611c0047e0 con 0x7f612c13f710 2024-03-25T19:31:13.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.634+0000 7f6121ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:13.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.635+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f611c001d80 con 0x7f612c13f710 2024-03-25T19:31:13.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.635+0000 7f6121ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:13.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.635+0000 7f6121ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:13.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.635+0000 7f6121ffb640 10 monclient: dump: 2024-03-25T19:31:13.635 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:13.635 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:13.635 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:13.635 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:13.636 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:13.636 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:13.636 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:13.636 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:13.636 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:13.636 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:13.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.635+0000 7f61237fe640 4 set_mon_vals no callback set 2024-03-25T19:31:13.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.635+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f611c004cb0 con 0x7f612c13f710 2024-03-25T19:31:13.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.636+0000 7f6131189640 10 monclient: get_auth_request con 0x7f611006dd10 auth_method 0 2024-03-25T19:31:13.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.636+0000 7f6121ffb640 10 client.4329.objecter ms_handle_connect 0x7f611006dd10 2024-03-25T19:31:13.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.646+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f611c0547e0 con 0x7f612c13f710 2024-03-25T19:31:13.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.646+0000 7f6121ffb640 10 client.4329.objecter ms_dispatch 0x7f612c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:13.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.646+0000 7f6121ffb640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:13.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.646+0000 7f6121ffb640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:13.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.646+0000 7f6121ffb640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-25T19:31:13.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.646+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f611c0502f0 con 0x7f612c13f710 2024-03-25T19:31:13.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.646+0000 7f6121ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:13.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.646+0000 7f6121ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:13.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.905+0000 7f61333ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:13.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.905+0000 7f61333ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:13.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.905+0000 7f61333ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:13.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.905+0000 7f61333ed640 1 -- 172.21.15.19:0/3309444359 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f612c0627b0 con 0x7f612c13f710 2024-03-25T19:31:13.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.930+0000 7f6121ffb640 1 -- 172.21.15.19:0/3309444359 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f611c052220 con 0x7f612c13f710 2024-03-25T19:31:13.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.931+0000 7f6121ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:13.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.931+0000 7f6121ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:13.931 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:31:13.931 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:13.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.933+0000 7f61333ed640 10 librados: watch_flush enter 2024-03-25T19:31:13.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.933+0000 7f61333ed640 10 librados: watch_flush exit 2024-03-25T19:31:13.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.933+0000 7f61333ed640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-25T19:31:13.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.933+0000 7f61333ed640 10 client.4329.objecter successfully canceled tick 2024-03-25T19:31:13.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.933+0000 7f61333ed640 1 -- 172.21.15.19:0/3309444359 >> v1:172.21.15.188:6800/2207506229 conn(0x7f611006dd10 legacy=0x7f61100701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.933+0000 7f61333ed640 10 monclient: shutdown 2024-03-25T19:31:13.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.933+0000 7f61333ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:13.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.933+0000 7f61333ed640 1 -- 172.21.15.19:0/3309444359 >> v1:172.21.15.19:6789/0 conn(0x7f612c13f710 legacy=0x7f612c0b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.933+0000 7f61333ed640 1 -- 172.21.15.19:0/3309444359 shutdown_connections 2024-03-25T19:31:13.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.933+0000 7f61333ed640 1 -- 172.21.15.19:0/3309444359 >> 172.21.15.19:0/3309444359 conn(0x7f612c0b7ea0 msgr2=0x7f612c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:13.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.934+0000 7f61333ed640 1 -- 172.21.15.19:0/3309444359 shutdown_connections 2024-03-25T19:31:13.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.934+0000 7f61333ed640 1 -- 172.21.15.19:0/3309444359 wait complete. 2024-03-25T19:31:13.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:13.934+0000 7f61333ed640 1 librados: shutdown 2024-03-25T19:31:13.947 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:20.754 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:20.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.870+0000 7fae2e40d640 10 monclient: get_monmap_and_config 2024-03-25T19:31:20.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.870+0000 7fae2e40d640 10 monclient: build_initial_monmap 2024-03-25T19:31:20.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.870+0000 7fae2e40d640 10 monclient: monmap: 2024-03-25T19:31:20.870 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:20.870 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:20.870 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:31:20.871274+0000 2024-03-25T19:31:20.870 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:31:20.871274+0000 2024-03-25T19:31:20.870 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:20.870 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:20.870 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:20.871 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:31:20.871 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:31:20.871 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:20.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.871+0000 7fae2e40d640 1 Processor -- start 2024-03-25T19:31:20.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.871+0000 7fae2e40d640 1 -- start start 2024-03-25T19:31:20.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.871+0000 7fae2e40d640 10 monclient: init 2024-03-25T19:31:20.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.872+0000 7fae2e40d640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:20.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.872+0000 7fae2e40d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:20.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.872+0000 7fae2e40d640 10 monclient(hunting): picked mon.noname-a con 0x7fae280a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:20.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.872+0000 7fae2e40d640 10 monclient(hunting): picked mon.noname-b con 0x7fae2813f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:20.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.872+0000 7fae2e40d640 10 monclient(hunting): picked mon.noname-c con 0x7fae281430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:20.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.872+0000 7fae2e40d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae28142520 con 0x7fae280a98f0 2024-03-25T19:31:20.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.872+0000 7fae2e40d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae280b28a0 con 0x7fae281430c0 2024-03-25T19:31:20.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.872+0000 7fae2e40d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae280b2a30 con 0x7fae2813f710 2024-03-25T19:31:20.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.872+0000 7fae2e40d640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:20.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.872+0000 7fae2e40d640 10 monclient(hunting): authenticate will time out at 6134.142254s 2024-03-25T19:31:20.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.880+0000 7fae2c9aa640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fae281430c0 0x7fae2814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:48500/0 (socket says 172.21.15.19:48500) 2024-03-25T19:31:20.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.880+0000 7fae2c9aa640 1 -- 172.21.15.19:0/4004281434 learned_addr learned my addr 172.21.15.19:0/4004281434 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:20.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.897+0000 7fae26ffd640 1 -- 172.21.15.19:0/4004281434 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169075025 0 0) 0x7fae280b28a0 con 0x7fae281430c0 2024-03-25T19:31:20.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.897+0000 7fae26ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:20.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.897+0000 7fae26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:20.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.897+0000 7fae26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:20.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.897+0000 7fae26ffd640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:31:20.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.897+0000 7fae26ffd640 1 -- 172.21.15.19:0/4004281434 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae10003520 con 0x7fae281430c0 2024-03-25T19:31:20.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.897+0000 7fae26ffd640 1 -- 172.21.15.19:0/4004281434 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926167351 0 0) 0x7fae28142520 con 0x7fae280a98f0 2024-03-25T19:31:20.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.897+0000 7fae26ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:20.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.897+0000 7fae26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:20.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.897+0000 7fae26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:20.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.897+0000 7fae26ffd640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:31:20.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.897+0000 7fae26ffd640 1 -- 172.21.15.19:0/4004281434 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae280b28a0 con 0x7fae280a98f0 2024-03-25T19:31:20.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 1 -- 172.21.15.19:0/4004281434 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2077794458 0 0) 0x7fae10003520 con 0x7fae281430c0 2024-03-25T19:31:20.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 1 -- 172.21.15.19:0/4004281434 >> v1:172.21.15.19:6790/0 conn(0x7fae2813f710 legacy=0x7fae281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 1 -- 172.21.15.19:0/4004281434 >> v1:172.21.15.19:6789/0 conn(0x7fae280a98f0 legacy=0x7fae280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 1 monclient: found mon.noname-c 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 1 -- 172.21.15.19:0/4004281434 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae280b44c0 con 0x7fae281430c0 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 10 monclient: _finish_auth 0 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 1 -- 172.21.15.19:0/4004281434 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fae18001690 con 0x7fae281430c0 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:20.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae26ffd640 10 monclient: dump: 2024-03-25T19:31:20.915 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:20.915 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:20.915 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:20.915 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:20.915 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:20.915 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:20.915 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:20.915 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:20.915 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:20.915 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:20.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae2e40d640 5 monclient: authenticate success, global_id 4327 2024-03-25T19:31:20.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.913+0000 7fae2e40d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:20.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.943+0000 7fae26ffd640 1 -- 172.21.15.19:0/4004281434 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae180017f0 con 0x7fae281430c0 2024-03-25T19:31:20.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.943+0000 7fae26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:20.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.943+0000 7fae26ffd640 1 -- 172.21.15.19:0/4004281434 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fae18001c00 con 0x7fae281430c0 2024-03-25T19:31:20.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.943+0000 7fae2e40d640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:20.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.943+0000 7fae2e40d640 4 set_mon_vals no callback set 2024-03-25T19:31:20.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.943+0000 7fae26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:20.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.943+0000 7fae26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:20.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.943+0000 7fae26ffd640 10 monclient: dump: 2024-03-25T19:31:20.944 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:20.944 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.944+0000 7fae2e40d640 10 monclient: shutdown 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.944+0000 7fae2e40d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.944+0000 7fae2e40d640 1 -- 172.21.15.19:0/4004281434 >> v1:172.21.15.188:6789/0 conn(0x7fae281430c0 legacy=0x7fae2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.944+0000 7fae2e40d640 1 -- 172.21.15.19:0/4004281434 shutdown_connections 2024-03-25T19:31:20.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.944+0000 7fae2e40d640 1 -- 172.21.15.19:0/4004281434 >> 172.21.15.19:0/4004281434 conn(0x7fae280b7ea0 msgr2=0x7fae280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:20.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.944+0000 7fae2e40d640 1 -- 172.21.15.19:0/4004281434 shutdown_connections 2024-03-25T19:31:20.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.944+0000 7fae2e40d640 1 -- 172.21.15.19:0/4004281434 wait complete. 2024-03-25T19:31:20.947 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.944+0000 7fae2e40d640 10 monclient: build_initial_monmap 2024-03-25T19:31:20.947 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.944+0000 7fae2e40d640 10 monclient: monmap: 2024-03-25T19:31:20.947 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.945+0000 7fae2e40d640 1 librados: starting msgr at 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.945+0000 7fae2e40d640 1 librados: starting objecter 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.945+0000 7fae2e40d640 1 Processor -- start 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.945+0000 7fae2e40d640 1 -- start start 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.945+0000 7fae2e40d640 1 librados: setting wanted keys 2024-03-25T19:31:20.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.945+0000 7fae2e40d640 1 librados: calling monclient init 2024-03-25T19:31:20.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.945+0000 7fae2e40d640 10 monclient: init 2024-03-25T19:31:20.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.946+0000 7fae2e40d640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:20.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.946+0000 7fae2e40d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:20.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.946+0000 7fae2e40d640 10 monclient(hunting): picked mon.noname-b con 0x7fae280a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:20.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.946+0000 7fae2e40d640 10 monclient(hunting): picked mon.noname-a con 0x7fae2813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:20.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.946+0000 7fae2e40d640 10 monclient(hunting): picked mon.noname-c con 0x7fae281430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:20.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.946+0000 7fae2e40d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2813c2b0 con 0x7fae2813f710 2024-03-25T19:31:20.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.946+0000 7fae2e40d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae28136f10 con 0x7fae280a98f0 2024-03-25T19:31:20.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.946+0000 7fae2e40d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae281370a0 con 0x7fae281430c0 2024-03-25T19:31:20.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.946+0000 7fae2e40d640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:20.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.946+0000 7fae2e40d640 10 monclient(hunting): authenticate will time out at 6134.216396s 2024-03-25T19:31:20.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.949+0000 7fae2c9aa640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fae281430c0 0x7fae28133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:42926/0 (socket says 172.21.15.19:42926) 2024-03-25T19:31:20.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.949+0000 7fae2c9aa640 1 -- 172.21.15.19:0/2211684418 learned_addr learned my addr 172.21.15.19:0/2211684418 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:20.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.950+0000 7fae24ff9640 10 client.?.objecter ms_handle_connect 0x7fae281430c0 2024-03-25T19:31:20.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.950+0000 7fae24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:20.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.954+0000 7fae24ff9640 10 client.?.objecter ms_handle_connect 0x7fae280a98f0 2024-03-25T19:31:20.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:20.954+0000 7fae24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:21.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.004+0000 7fae24ff9640 10 client.?.objecter ms_handle_connect 0x7fae2813f710 2024-03-25T19:31:21.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.004+0000 7fae24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:21.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.017+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170910487 0 0) 0x7fae2813c2b0 con 0x7fae2813f710 2024-03-25T19:31:21.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.017+0000 7fae24ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:21.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.017+0000 7fae24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:21.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.017+0000 7fae24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:21.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.017+0000 7fae24ff9640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:31:21.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.017+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae14004eb0 con 0x7fae2813f710 2024-03-25T19:31:21.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.041+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1949135313 0 0) 0x7fae14004eb0 con 0x7fae2813f710 2024-03-25T19:31:21.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.041+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae2813c2b0 con 0x7fae2813f710 2024-03-25T19:31:21.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.041+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fae1c0018c0 con 0x7fae2813f710 2024-03-25T19:31:21.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.041+0000 7fae24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:21.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.041+0000 7fae24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:21.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.041+0000 7fae24ff9640 10 monclient(hunting): dump: 2024-03-25T19:31:21.041 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:21.041 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:21.041 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:21.041 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:21.041 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:21.042 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:21.042 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:21.042 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:21.042 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:21.042 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:21.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.041+0000 7fae24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:21.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 202501357 0 0) 0x7fae2813c2b0 con 0x7fae2813f710 2024-03-25T19:31:21.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 >> v1:172.21.15.19:6790/0 conn(0x7fae281430c0 legacy=0x7fae28133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:21.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 >> v1:172.21.15.188:6789/0 conn(0x7fae280a98f0 legacy=0x7fae281d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:21.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae24ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:31:21.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae24ff9640 1 monclient: found mon.a 2024-03-25T19:31:21.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:21.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae24ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:21.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae28137230 con 0x7fae2813f710 2024-03-25T19:31:21.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae24ff9640 10 monclient: _finish_auth 0 2024-03-25T19:31:21.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae24ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:31:21.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:21.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae2e40d640 5 monclient: authenticate success, global_id 4335 2024-03-25T19:31:21.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae2e40d640 10 monclient: _renew_subs 2024-03-25T19:31:21.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae2e40d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:21.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae2e40d640 1 -- 172.21.15.19:0/2211684418 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2813b620 con 0x7fae2813f710 2024-03-25T19:31:21.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae2e40d640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:21.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae2e40d640 10 monclient: _renew_subs 2024-03-25T19:31:21.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae2e40d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:21.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae2e40d640 1 -- 172.21.15.19:0/2211684418 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae281338f0 con 0x7fae2813f710 2024-03-25T19:31:21.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.054+0000 7fae2e40d640 1 librados: init done 2024-03-25T19:31:21.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.055+0000 7fae2e40d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:21.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.055+0000 7fae2e40d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:21.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.055+0000 7fae2e40d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:21.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.055+0000 7fae2e40d640 1 -- 172.21.15.19:0/2211684418 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2813b9c0 con 0x7fae2813f710 2024-03-25T19:31:21.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.401+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae1c0035f0 con 0x7fae2813f710 2024-03-25T19:31:21.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.401+0000 7fae24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:21.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.401+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fae1c003820 con 0x7fae2813f710 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.401+0000 7fae24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.401+0000 7fae24ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.401+0000 7fae24ff9640 10 monclient: dump: 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:21.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.401+0000 7fae267fc640 4 set_mon_vals no callback set 2024-03-25T19:31:21.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.401+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fae1c003a40 con 0x7fae2813f710 2024-03-25T19:31:21.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.402+0000 7fae27fff640 10 monclient: get_auth_request con 0x7fae1406dce0 auth_method 0 2024-03-25T19:31:21.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.402+0000 7fae24ff9640 10 client.4335.objecter ms_handle_connect 0x7fae1406dce0 2024-03-25T19:31:21.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.409+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fae1c053420 con 0x7fae2813f710 2024-03-25T19:31:21.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.409+0000 7fae24ff9640 10 client.4335.objecter ms_dispatch 0x7fae28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:21.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.409+0000 7fae24ff9640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:21.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.409+0000 7fae24ff9640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:21.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.409+0000 7fae24ff9640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-25T19:31:21.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.409+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae1c04ef30 con 0x7fae2813f710 2024-03-25T19:31:21.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.409+0000 7fae24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:21.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.409+0000 7fae24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:21.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.674+0000 7fae2e40d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:21.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.674+0000 7fae2e40d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:21.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.674+0000 7fae2e40d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:21.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.674+0000 7fae2e40d640 1 -- 172.21.15.19:0/2211684418 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae280627b0 con 0x7fae2813f710 2024-03-25T19:31:21.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.697+0000 7fae24ff9640 1 -- 172.21.15.19:0/2211684418 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fae1c050e60 con 0x7fae2813f710 2024-03-25T19:31:21.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.697+0000 7fae24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:21.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.697+0000 7fae24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:21.698 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:31:21.698 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:21.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 10 librados: watch_flush enter 2024-03-25T19:31:21.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 10 librados: watch_flush exit 2024-03-25T19:31:21.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-25T19:31:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 10 client.4335.objecter successfully canceled tick 2024-03-25T19:31:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 1 -- 172.21.15.19:0/2211684418 >> v1:172.21.15.188:6800/2207506229 conn(0x7fae1406dce0 legacy=0x7fae140701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 10 monclient: shutdown 2024-03-25T19:31:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 1 -- 172.21.15.19:0/2211684418 >> v1:172.21.15.19:6789/0 conn(0x7fae2813f710 legacy=0x7fae281377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 1 -- 172.21.15.19:0/2211684418 shutdown_connections 2024-03-25T19:31:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 1 -- 172.21.15.19:0/2211684418 >> 172.21.15.19:0/2211684418 conn(0x7fae280b7ea0 msgr2=0x7fae28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 1 -- 172.21.15.19:0/2211684418 shutdown_connections 2024-03-25T19:31:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 1 -- 172.21.15.19:0/2211684418 wait complete. 2024-03-25T19:31:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:21.700+0000 7fae2e40d640 1 librados: shutdown 2024-03-25T19:31:21.712 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:28.518 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:28.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.635+0000 7fd1aa2d3640 10 monclient: get_monmap_and_config 2024-03-25T19:31:28.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.635+0000 7fd1aa2d3640 10 monclient: build_initial_monmap 2024-03-25T19:31:28.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.635+0000 7fd1aa2d3640 10 monclient: monmap: 2024-03-25T19:31:28.635 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:28.635 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:28.635 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:31:28.636681+0000 2024-03-25T19:31:28.636 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:31:28.636681+0000 2024-03-25T19:31:28.636 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:28.636 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:28.636 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:28.636 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:31:28.636 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:31:28.636 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:28.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.636+0000 7fd1aa2d3640 1 Processor -- start 2024-03-25T19:31:28.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.636+0000 7fd1aa2d3640 1 -- start start 2024-03-25T19:31:28.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.636+0000 7fd1aa2d3640 10 monclient: init 2024-03-25T19:31:28.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.637+0000 7fd1aa2d3640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:28.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.637+0000 7fd1aa2d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:28.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.637+0000 7fd1aa2d3640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a40b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:28.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.637+0000 7fd1aa2d3640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a40b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:28.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.637+0000 7fd1aa2d3640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a414be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:28.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.637+0000 7fd1aa2d3640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a40af940 con 0x7fd1a40b1730 2024-03-25T19:31:28.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.637+0000 7fd1aa2d3640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a40b0a70 con 0x7fd1a414be90 2024-03-25T19:31:28.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.637+0000 7fd1aa2d3640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a40b0d50 con 0x7fd1a40b4d60 2024-03-25T19:31:28.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.637+0000 7fd1aa2d3640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:28.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.637+0000 7fd1aa2d3640 10 monclient(hunting): authenticate will time out at 6141.907664s 2024-03-25T19:31:28.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.647+0000 7fd1a3fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fd1a40b4d60 0x7fd1a413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:48046/0 (socket says 172.21.15.19:48046) 2024-03-25T19:31:28.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.647+0000 7fd1a3fff640 1 -- 172.21.15.19:0/406172120 learned_addr learned my addr 172.21.15.19:0/406172120 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:28.655 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.655+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/406172120 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823647491 0 0) 0x7fd1a40b0d50 con 0x7fd1a40b4d60 2024-03-25T19:31:28.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.655+0000 7fd1a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:28.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.655+0000 7fd1a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:28.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.655+0000 7fd1a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:28.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.655+0000 7fd1a2ffd640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:31:28.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.655+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/406172120 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd18c003520 con 0x7fd1a40b4d60 2024-03-25T19:31:28.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.663+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/406172120 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100447224 0 0) 0x7fd1a40af940 con 0x7fd1a40b1730 2024-03-25T19:31:28.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.663+0000 7fd1a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:28.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.663+0000 7fd1a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:28.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.663+0000 7fd1a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:28.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.663+0000 7fd1a2ffd640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:31:28.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.663+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/406172120 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a40b0d50 con 0x7fd1a40b1730 2024-03-25T19:31:28.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/406172120 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 693142034 0 0) 0x7fd1a40b0d50 con 0x7fd1a40b1730 2024-03-25T19:31:28.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/406172120 >> v1:172.21.15.19:6790/0 conn(0x7fd1a40b4d60 legacy=0x7fd1a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/406172120 >> v1:172.21.15.188:6789/0 conn(0x7fd1a414be90 legacy=0x7fd1a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:28.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 1 monclient: found mon.noname-a 2024-03-25T19:31:28.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:28.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:31:28.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/406172120 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a40b1030 con 0x7fd1a40b1730 2024-03-25T19:31:28.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:31:28.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:28.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:28.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/406172120 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd198001670 con 0x7fd1a40b1730 2024-03-25T19:31:28.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:28.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:28.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1a2ffd640 10 monclient: dump: 2024-03-25T19:31:28.675 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:28.675 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:28.675 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:28.675 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:28.675 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:28.675 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:28.675 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:28.675 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:28.675 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:28.675 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:28.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1aa2d3640 5 monclient: authenticate success, global_id 4338 2024-03-25T19:31:28.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.673+0000 7fd1aa2d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:28.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.686+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/406172120 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1980017d0 con 0x7fd1a40b1730 2024-03-25T19:31:28.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.686+0000 7fd1a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:28.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.686+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/406172120 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd198001ba0 con 0x7fd1a40b1730 2024-03-25T19:31:28.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.686+0000 7fd1aa2d3640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:28.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.686+0000 7fd1aa2d3640 4 set_mon_vals no callback set 2024-03-25T19:31:28.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.686+0000 7fd1a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:28.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.686+0000 7fd1a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:28.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.686+0000 7fd1a2ffd640 10 monclient: dump: 2024-03-25T19:31:28.687 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:28.687 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:28.687 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:28.687 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:28.687 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:28.688 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:28.688 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:28.688 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:28.688 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:28.688 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:28.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.687+0000 7fd1aa2d3640 10 monclient: shutdown 2024-03-25T19:31:28.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.687+0000 7fd1aa2d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:28.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.687+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/406172120 >> v1:172.21.15.19:6789/0 conn(0x7fd1a40b1730 legacy=0x7fd1a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.687+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/406172120 shutdown_connections 2024-03-25T19:31:28.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.687+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/406172120 >> 172.21.15.19:0/406172120 conn(0x7fd1a40617a0 msgr2=0x7fd1a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:28.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.687+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/406172120 shutdown_connections 2024-03-25T19:31:28.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.688+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/406172120 wait complete. 2024-03-25T19:31:28.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.688+0000 7fd1aa2d3640 10 monclient: build_initial_monmap 2024-03-25T19:31:28.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.688+0000 7fd1aa2d3640 10 monclient: monmap: 2024-03-25T19:31:28.691 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:28.691 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:28.691 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:31:28.691 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:31:28.691 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:28.691 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:31:28.691 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:28.691 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:31:28.691 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:31:28.691 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.689+0000 7fd1aa2d3640 1 librados: starting msgr at 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.689+0000 7fd1aa2d3640 1 librados: starting objecter 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.689+0000 7fd1aa2d3640 1 Processor -- start 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.689+0000 7fd1aa2d3640 1 -- start start 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.689+0000 7fd1aa2d3640 1 librados: setting wanted keys 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.689+0000 7fd1aa2d3640 1 librados: calling monclient init 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.689+0000 7fd1aa2d3640 10 monclient: init 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.690+0000 7fd1aa2d3640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.690+0000 7fd1aa2d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.690+0000 7fd1aa2d3640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a40b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.690+0000 7fd1aa2d3640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a40b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.690+0000 7fd1aa2d3640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a414be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:28.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.690+0000 7fd1aa2d3640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a413ab60 con 0x7fd1a40b4d60 2024-03-25T19:31:28.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.690+0000 7fd1aa2d3640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a4137040 con 0x7fd1a40b1730 2024-03-25T19:31:28.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.690+0000 7fd1aa2d3640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a413bb80 con 0x7fd1a414be90 2024-03-25T19:31:28.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.690+0000 7fd1aa2d3640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:28.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.690+0000 7fd1aa2d3640 10 monclient(hunting): authenticate will time out at 6141.960242s 2024-03-25T19:31:28.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.701+0000 7fd1a8870640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fd1a414be90 0x7fd1a4136930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:48108/0 (socket says 172.21.15.19:48108) 2024-03-25T19:31:28.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.701+0000 7fd1a8870640 1 -- 172.21.15.19:0/44592908 learned_addr learned my addr 172.21.15.19:0/44592908 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:28.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.702+0000 7fd1a0ff9640 10 client.?.objecter ms_handle_connect 0x7fd1a414be90 2024-03-25T19:31:28.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.702+0000 7fd1a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:28.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.712+0000 7fd1a0ff9640 10 client.?.objecter ms_handle_connect 0x7fd1a40b1730 2024-03-25T19:31:28.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.712+0000 7fd1a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:28.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.715+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519528619 0 0) 0x7fd1a413bb80 con 0x7fd1a414be90 2024-03-25T19:31:28.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.715+0000 7fd1a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:28.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.715+0000 7fd1a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:28.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.715+0000 7fd1a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:28.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.715+0000 7fd1a0ff9640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:31:28.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.715+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd190004eb0 con 0x7fd1a414be90 2024-03-25T19:31:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.717+0000 7fd1a0ff9640 10 client.?.objecter ms_handle_connect 0x7fd1a40b4d60 2024-03-25T19:31:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.717+0000 7fd1a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:28.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.726+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354241467 0 0) 0x7fd1a413ab60 con 0x7fd1a40b4d60 2024-03-25T19:31:28.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.726+0000 7fd1a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:28.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.726+0000 7fd1a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:28.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.726+0000 7fd1a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:28.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.726+0000 7fd1a0ff9640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:31:28.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.726+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a413bb80 con 0x7fd1a40b4d60 2024-03-25T19:31:28.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.727+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 413776441 0 0) 0x7fd190004eb0 con 0x7fd1a414be90 2024-03-25T19:31:28.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.727+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1a413ab60 con 0x7fd1a414be90 2024-03-25T19:31:28.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.727+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd19c001c30 con 0x7fd1a414be90 2024-03-25T19:31:28.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.727+0000 7fd1a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:28.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.727+0000 7fd1a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:28.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.727+0000 7fd1a0ff9640 10 monclient(hunting): dump: 2024-03-25T19:31:28.727 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:28.728 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:28.728 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:28.728 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:28.728 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:28.728 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:28.728 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:28.728 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:28.728 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:28.728 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:28.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.727+0000 7fd1a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:28.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.728+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292622018 0 0) 0x7fd1a4137040 con 0x7fd1a40b1730 2024-03-25T19:31:28.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.728+0000 7fd1a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:28.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.728+0000 7fd1a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:28.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.728+0000 7fd1a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:28.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.728+0000 7fd1a0ff9640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:31:28.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.728+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd190004eb0 con 0x7fd1a40b1730 2024-03-25T19:31:28.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2471982232 0 0) 0x7fd1a413ab60 con 0x7fd1a414be90 2024-03-25T19:31:28.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 >> v1:172.21.15.188:6789/0 conn(0x7fd1a40b1730 legacy=0x7fd1a4141c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 >> v1:172.21.15.19:6789/0 conn(0x7fd1a40b4d60 legacy=0x7fd1a4136220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1a0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:31:28.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1a0ff9640 1 monclient: found mon.c 2024-03-25T19:31:28.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:28.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1a0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:31:28.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a413be60 con 0x7fd1a414be90 2024-03-25T19:31:28.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1a0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:31:28.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1a0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:31:28.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:28.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1aa2d3640 5 monclient: authenticate success, global_id 4313 2024-03-25T19:31:28.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1aa2d3640 10 monclient: _renew_subs 2024-03-25T19:31:28.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1aa2d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:31:28.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/44592908 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a406df20 con 0x7fd1a414be90 2024-03-25T19:31:28.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1aa2d3640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:28.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1aa2d3640 10 monclient: _renew_subs 2024-03-25T19:31:28.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1aa2d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:31:28.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/44592908 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a413aff0 con 0x7fd1a414be90 2024-03-25T19:31:28.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.740+0000 7fd1aa2d3640 1 librados: init done 2024-03-25T19:31:28.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.741+0000 7fd1aa2d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:28.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.741+0000 7fd1aa2d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:28.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.741+0000 7fd1aa2d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:31:28.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:28.741+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/44592908 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a40b3b40 con 0x7fd1a414be90 2024-03-25T19:31:29.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.063+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd19c003990 con 0x7fd1a414be90 2024-03-25T19:31:29.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.063+0000 7fd1a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:29.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.063+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd19c003da0 con 0x7fd1a414be90 2024-03-25T19:31:29.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.063+0000 7fd1a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:29.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.063+0000 7fd1a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:29.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.063+0000 7fd1a0ff9640 10 monclient: dump: 2024-03-25T19:31:29.063 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:29.063 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:29.064 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:29.064 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:29.064 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:29.064 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:29.064 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:29.064 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:29.064 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:29.064 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:29.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.063+0000 7fd1a27fc640 4 set_mon_vals no callback set 2024-03-25T19:31:29.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.064+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fd19c003fc0 con 0x7fd1a414be90 2024-03-25T19:31:29.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.064+0000 7fd1a3fff640 10 monclient: get_auth_request con 0x7fd19006de00 auth_method 0 2024-03-25T19:31:29.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.065+0000 7fd1a0ff9640 10 client.4313.objecter ms_handle_connect 0x7fd19006de00 2024-03-25T19:31:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.070+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fd19c0549c0 con 0x7fd1a414be90 2024-03-25T19:31:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.070+0000 7fd1a0ff9640 10 client.4313.objecter ms_dispatch 0x7fd1a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.070+0000 7fd1a0ff9640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.070+0000 7fd1a0ff9640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.070+0000 7fd1a0ff9640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-25T19:31:29.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.070+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd19c0504d0 con 0x7fd1a414be90 2024-03-25T19:31:29.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.070+0000 7fd1a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:29.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.070+0000 7fd1a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:29.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.321+0000 7fd1aa2d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:29.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.321+0000 7fd1aa2d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:29.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.321+0000 7fd1aa2d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:31:29.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.321+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/44592908 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1a40627b0 con 0x7fd1a414be90 2024-03-25T19:31:29.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.456+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/44592908 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fd19c052400 con 0x7fd1a414be90 2024-03-25T19:31:29.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.456+0000 7fd1a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:29.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.456+0000 7fd1a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:29.457 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:31:29.457 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:29.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 10 librados: watch_flush enter 2024-03-25T19:31:29.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 10 librados: watch_flush exit 2024-03-25T19:31:29.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-25T19:31:29.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 10 client.4313.objecter successfully canceled tick 2024-03-25T19:31:29.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/44592908 >> v1:172.21.15.188:6800/2207506229 conn(0x7fd19006de00 legacy=0x7fd1900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:29.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 10 monclient: shutdown 2024-03-25T19:31:29.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:29.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/44592908 >> v1:172.21.15.19:6790/0 conn(0x7fd1a414be90 legacy=0x7fd1a4136930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:29.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/44592908 shutdown_connections 2024-03-25T19:31:29.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/44592908 >> 172.21.15.19:0/44592908 conn(0x7fd1a40617a0 msgr2=0x7fd1a40b63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:29.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/44592908 shutdown_connections 2024-03-25T19:31:29.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 1 -- 172.21.15.19:0/44592908 wait complete. 2024-03-25T19:31:29.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:29.458+0000 7fd1aa2d3640 1 librados: shutdown 2024-03-25T19:31:29.470 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:32.028 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:54.758 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:32.079 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:54.809 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:32.108 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:54.837 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:32.127 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:02:54.830 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:32.131 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:02:54.834 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:32.146 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:54.876 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:32.156 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:54.885 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:32.174 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:02:54.877 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:32.186 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:54.916 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:36.277 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:36.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.397+0000 7f770ce17640 10 monclient: get_monmap_and_config 2024-03-25T19:31:36.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.397+0000 7f770ce17640 10 monclient: build_initial_monmap 2024-03-25T19:31:36.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.397+0000 7f770ce17640 10 monclient: monmap: 2024-03-25T19:31:36.397 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:36.398 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:36.398 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:31:36.398805+0000 2024-03-25T19:31:36.398 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:31:36.398805+0000 2024-03-25T19:31:36.398 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:36.398 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:36.398 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:36.398 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:31:36.398 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:31:36.398 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:36.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.398+0000 7f770ce17640 1 Processor -- start 2024-03-25T19:31:36.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.398+0000 7f770ce17640 1 -- start start 2024-03-25T19:31:36.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.398+0000 7f770ce17640 10 monclient: init 2024-03-25T19:31:36.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.399+0000 7f770ce17640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:36.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.399+0000 7f770ce17640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:36.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.399+0000 7f770ce17640 10 monclient(hunting): picked mon.noname-a con 0x7f77080b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:36.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.399+0000 7f770ce17640 10 monclient(hunting): picked mon.noname-b con 0x7f77080b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:36.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.399+0000 7f770ce17640 10 monclient(hunting): picked mon.noname-c con 0x7f770814be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:36.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.399+0000 7f770ce17640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080af940 con 0x7f77080b4d60 2024-03-25T19:31:36.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.399+0000 7f770ce17640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080b0a70 con 0x7f770814be90 2024-03-25T19:31:36.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.399+0000 7f770ce17640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080b0d50 con 0x7f77080b1730 2024-03-25T19:31:36.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.399+0000 7f770ce17640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:36.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.399+0000 7f770ce17640 10 monclient(hunting): authenticate will time out at 6149.669521s 2024-03-25T19:31:36.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.408+0000 7f770659c640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f77080b4d60 0x7f770813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:50504/0 (socket says 172.21.15.19:50504) 2024-03-25T19:31:36.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.408+0000 7f770659c640 1 -- 172.21.15.19:0/3688443258 learned_addr learned my addr 172.21.15.19:0/3688443258 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:36.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.420+0000 7f770559a640 1 -- 172.21.15.19:0/3688443258 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117345958 0 0) 0x7f77080af940 con 0x7f77080b4d60 2024-03-25T19:31:36.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.420+0000 7f770559a640 10 monclient(hunting): _negotiate 2024-03-25T19:31:36.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.420+0000 7f770559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:36.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.420+0000 7f770559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:36.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.420+0000 7f770559a640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:31:36.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.420+0000 7f770559a640 1 -- 172.21.15.19:0/3688443258 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f0003520 con 0x7f77080b4d60 2024-03-25T19:31:36.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.429+0000 7f770559a640 1 -- 172.21.15.19:0/3688443258 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847592556 0 0) 0x7f77080b0a70 con 0x7f770814be90 2024-03-25T19:31:36.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.429+0000 7f770559a640 10 monclient(hunting): _negotiate 2024-03-25T19:31:36.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.429+0000 7f770559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:36.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.429+0000 7f770559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:36.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.429+0000 7f770559a640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:31:36.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.429+0000 7f770559a640 1 -- 172.21.15.19:0/3688443258 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77080af940 con 0x7f770814be90 2024-03-25T19:31:36.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 1 -- 172.21.15.19:0/3688443258 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2533822994 0 0) 0x7f76f0003520 con 0x7f77080b4d60 2024-03-25T19:31:36.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 1 -- 172.21.15.19:0/3688443258 >> v1:172.21.15.19:6790/0 conn(0x7f77080b1730 legacy=0x7f77080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 1 -- 172.21.15.19:0/3688443258 >> v1:172.21.15.188:6789/0 conn(0x7f770814be90 legacy=0x7f770814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 10 monclient: _finish_hunting 0 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 1 monclient: found mon.noname-a 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 1 -- 172.21.15.19:0/3688443258 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080b1030 con 0x7f77080b4d60 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 10 monclient: _finish_auth 0 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 10 monclient: _check_auth_tickets 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 1 -- 172.21.15.19:0/3688443258 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f76f8001690 con 0x7f77080b4d60 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770559a640 10 monclient: dump: 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:36.431 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:36.432 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:36.432 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:36.432 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:36.432 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:36.432 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:36.432 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:36.432 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:36.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770ce17640 5 monclient: authenticate success, global_id 4344 2024-03-25T19:31:36.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.430+0000 7f770ce17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:36.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.439+0000 7f770559a640 1 -- 172.21.15.19:0/3688443258 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76f80018b0 con 0x7f77080b4d60 2024-03-25T19:31:36.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.439+0000 7f770559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.439+0000 7f770559a640 1 -- 172.21.15.19:0/3688443258 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f76f8001c40 con 0x7f77080b4d60 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.439+0000 7f770559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.439+0000 7f770559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.439+0000 7f770559a640 10 monclient: dump: 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:36.440 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:36.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.439+0000 7f770ce17640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:36.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.439+0000 7f770ce17640 4 set_mon_vals no callback set 2024-03-25T19:31:36.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.440+0000 7f770ce17640 10 monclient: shutdown 2024-03-25T19:31:36.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.440+0000 7f770ce17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:36.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.440+0000 7f770ce17640 1 -- 172.21.15.19:0/3688443258 >> v1:172.21.15.19:6789/0 conn(0x7f77080b4d60 legacy=0x7f770813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.440+0000 7f770ce17640 1 -- 172.21.15.19:0/3688443258 shutdown_connections 2024-03-25T19:31:36.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.440+0000 7f770ce17640 1 -- 172.21.15.19:0/3688443258 >> 172.21.15.19:0/3688443258 conn(0x7f77080617a0 msgr2=0x7f7708061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:36.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.440+0000 7f770ce17640 1 -- 172.21.15.19:0/3688443258 shutdown_connections 2024-03-25T19:31:36.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.441+0000 7f770ce17640 1 -- 172.21.15.19:0/3688443258 wait complete. 2024-03-25T19:31:36.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.441+0000 7f770ce17640 10 monclient: build_initial_monmap 2024-03-25T19:31:36.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.441+0000 7f770ce17640 10 monclient: monmap: 2024-03-25T19:31:36.443 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.442+0000 7f770ce17640 1 librados: starting msgr at 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.442+0000 7f770ce17640 1 librados: starting objecter 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.442+0000 7f770ce17640 1 Processor -- start 2024-03-25T19:31:36.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.442+0000 7f770ce17640 1 -- start start 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.442+0000 7f770ce17640 1 librados: setting wanted keys 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.442+0000 7f770ce17640 1 librados: calling monclient init 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.442+0000 7f770ce17640 10 monclient: init 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.442+0000 7f770ce17640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.442+0000 7f770ce17640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.443+0000 7f770ce17640 10 monclient(hunting): picked mon.noname-a con 0x7f77080b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.443+0000 7f770ce17640 10 monclient(hunting): picked mon.noname-b con 0x7f77080b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.443+0000 7f770ce17640 10 monclient(hunting): picked mon.noname-c con 0x7f770814be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.443+0000 7f770ce17640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7708138210 con 0x7f77080b1730 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.443+0000 7f770ce17640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77081375a0 con 0x7f77080b4d60 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.443+0000 7f770ce17640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7708138440 con 0x7f770814be90 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.443+0000 7f770ce17640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:36.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.443+0000 7f770ce17640 10 monclient(hunting): authenticate will time out at 6149.713126s 2024-03-25T19:31:36.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.455+0000 7f7706d9d640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f770814be90 0x7f77081e3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:38410/0 (socket says 172.21.15.19:38410) 2024-03-25T19:31:36.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.455+0000 7f7706d9d640 1 -- 172.21.15.19:0/150218236 learned_addr learned my addr 172.21.15.19:0/150218236 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:36.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.456+0000 7f76f6ffd640 10 client.?.objecter ms_handle_connect 0x7f770814be90 2024-03-25T19:31:36.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.456+0000 7f76f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:36.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.461+0000 7f76f6ffd640 10 client.?.objecter ms_handle_connect 0x7f77080b1730 2024-03-25T19:31:36.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.461+0000 7f76f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:36.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.467+0000 7f76f6ffd640 10 client.?.objecter ms_handle_connect 0x7f77080b4d60 2024-03-25T19:31:36.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.467+0000 7f76f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:36.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.473+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292881936 0 0) 0x7f7708138440 con 0x7f770814be90 2024-03-25T19:31:36.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.473+0000 7f76f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:36.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.473+0000 7f76f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:36.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.473+0000 7f76f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:36.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.473+0000 7f76f6ffd640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:31:36.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.473+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e8004eb0 con 0x7f770814be90 2024-03-25T19:31:36.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.475+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395664545 0 0) 0x7f7708138210 con 0x7f77080b1730 2024-03-25T19:31:36.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.475+0000 7f76f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:36.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.475+0000 7f76f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:36.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.475+0000 7f76f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:36.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.475+0000 7f76f6ffd640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:31:36.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.475+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7708138440 con 0x7f77080b1730 2024-03-25T19:31:36.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.483+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2824602879 0 0) 0x7f7708138440 con 0x7f77080b1730 2024-03-25T19:31:36.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.483+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7708138210 con 0x7f77080b1730 2024-03-25T19:31:36.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.483+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f76f80015d0 con 0x7f77080b1730 2024-03-25T19:31:36.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.483+0000 7f76f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:36.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.483+0000 7f76f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:36.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.483+0000 7f76f6ffd640 10 monclient(hunting): dump: 2024-03-25T19:31:36.483 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:36.483 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:36.484 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:36.484 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:36.484 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:36.484 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:36.484 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:36.484 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:36.484 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:36.484 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:36.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.483+0000 7f76f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:36.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.485+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172010787 0 0) 0x7f77081375a0 con 0x7f77080b4d60 2024-03-25T19:31:36.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.485+0000 7f76f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:36.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.485+0000 7f76f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:36.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.485+0000 7f76f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:36.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.485+0000 7f76f6ffd640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:31:36.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.485+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7708138440 con 0x7f77080b4d60 2024-03-25T19:31:36.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.495+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1449840853 0 0) 0x7f76e8004eb0 con 0x7f770814be90 2024-03-25T19:31:36.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.495+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77081375a0 con 0x7f770814be90 2024-03-25T19:31:36.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.495+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7700001c90 con 0x7f770814be90 2024-03-25T19:31:36.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.495+0000 7f76f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:36.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.495+0000 7f76f6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:36.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.495+0000 7f76f6ffd640 10 monclient(hunting): dump: 2024-03-25T19:31:36.495 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:36.495 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:36.495 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:36.495 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:36.495 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:36.495 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:36.496 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:36.496 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:36.496 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:36.496 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:36.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.500+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 228234971 0 0) 0x7f7708138440 con 0x7f77080b4d60 2024-03-25T19:31:36.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.500+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76e8004eb0 con 0x7f77080b4d60 2024-03-25T19:31:36.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.500+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f76fc001830 con 0x7f77080b4d60 2024-03-25T19:31:36.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.500+0000 7f76f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:36.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.500+0000 7f76f6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:36.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.500+0000 7f76f6ffd640 10 monclient(hunting): dump: 2024-03-25T19:31:36.500 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:36.500 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:36.500 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:36.500 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:36.500 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:36.501 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:36.501 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:36.501 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:36.501 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:36.501 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:36.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.500+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3873186093 0 0) 0x7f7708138210 con 0x7f77080b1730 2024-03-25T19:31:36.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 >> v1:172.21.15.19:6790/0 conn(0x7f770814be90 legacy=0x7f77081e3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 >> v1:172.21.15.188:6789/0 conn(0x7f77080b4d60 legacy=0x7f770812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f76f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:36.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f76f6ffd640 1 monclient: found mon.a 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f76f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f76f6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77081364b0 con 0x7f77080b1730 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f76f6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f76f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f76f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f770ce17640 5 monclient: authenticate success, global_id 4347 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f770ce17640 10 monclient: _renew_subs 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f770ce17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f770ce17640 1 -- 172.21.15.19:0/150218236 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770813b470 con 0x7f77080b1730 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f770ce17640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f770ce17640 10 monclient: _renew_subs 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f770ce17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f770ce17640 1 -- 172.21.15.19:0/150218236 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7708136230 con 0x7f77080b1730 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.501+0000 7f770ce17640 1 librados: init done 2024-03-25T19:31:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.502+0000 7f770ce17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:36.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.502+0000 7f770ce17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:36.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.502+0000 7f770ce17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:36.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.502+0000 7f770ce17640 1 -- 172.21.15.19:0/150218236 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77081366f0 con 0x7f77080b1730 2024-03-25T19:31:36.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.824+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76f8004a90 con 0x7f77080b1730 2024-03-25T19:31:36.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.824+0000 7f76f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:36.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.824+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f76f8001750 con 0x7f77080b1730 2024-03-25T19:31:36.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.824+0000 7f76f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:36.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.824+0000 7f76f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:36.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.824+0000 7f76f6ffd640 10 monclient: dump: 2024-03-25T19:31:36.825 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:36.825 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:36.825 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:36.825 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:36.825 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:36.825 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:36.825 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:36.826 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:36.826 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:36.826 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:36.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.824+0000 7f7704d99640 4 set_mon_vals no callback set 2024-03-25T19:31:36.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.825+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f76f801ac10 con 0x7f77080b1730 2024-03-25T19:31:36.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.826+0000 7f7705d9b640 10 monclient: get_auth_request con 0x7f76e806e110 auth_method 0 2024-03-25T19:31:36.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.826+0000 7f76f6ffd640 10 client.4347.objecter ms_handle_connect 0x7f76e806e110 2024-03-25T19:31:36.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.831+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f76f80547e0 con 0x7f77080b1730 2024-03-25T19:31:36.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.831+0000 7f76f6ffd640 10 client.4347.objecter ms_dispatch 0x7f7708000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:36.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.831+0000 7f76f6ffd640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:36.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.831+0000 7f76f6ffd640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:36.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.831+0000 7f76f6ffd640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-25T19:31:36.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.831+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76f8050120 con 0x7f77080b1730 2024-03-25T19:31:36.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.831+0000 7f76f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:36.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:36.831+0000 7f76f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:37.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.050+0000 7f770ce17640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:37.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.050+0000 7f770ce17640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:37.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.050+0000 7f770ce17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:31:37.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.050+0000 7f770ce17640 1 -- 172.21.15.19:0/150218236 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77080627b0 con 0x7f77080b1730 2024-03-25T19:31:37.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.082+0000 7f76f6ffd640 1 -- 172.21.15.19:0/150218236 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f76f8052220 con 0x7f77080b1730 2024-03-25T19:31:37.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.082+0000 7f76f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:37.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.082+0000 7f76f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:37.083 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:31:37.083 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:37.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.084+0000 7f770ce17640 10 librados: watch_flush enter 2024-03-25T19:31:37.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.084+0000 7f770ce17640 10 librados: watch_flush exit 2024-03-25T19:31:37.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.084+0000 7f770ce17640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-25T19:31:37.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.084+0000 7f770ce17640 10 client.4347.objecter successfully canceled tick 2024-03-25T19:31:37.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.084+0000 7f770ce17640 1 -- 172.21.15.19:0/150218236 >> v1:172.21.15.188:6800/2207506229 conn(0x7f76e806e110 legacy=0x7f76e80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:37.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.084+0000 7f770ce17640 10 monclient: shutdown 2024-03-25T19:31:37.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.085+0000 7f770ce17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:37.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.085+0000 7f770ce17640 1 -- 172.21.15.19:0/150218236 >> v1:172.21.15.19:6789/0 conn(0x7f77080b1730 legacy=0x7f7708139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:37.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.085+0000 7f770ce17640 1 -- 172.21.15.19:0/150218236 shutdown_connections 2024-03-25T19:31:37.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.085+0000 7f770ce17640 1 -- 172.21.15.19:0/150218236 >> 172.21.15.19:0/150218236 conn(0x7f77080617a0 msgr2=0x7f770813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:37.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.085+0000 7f770ce17640 1 -- 172.21.15.19:0/150218236 shutdown_connections 2024-03-25T19:31:37.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.085+0000 7f770ce17640 1 -- 172.21.15.19:0/150218236 wait complete. 2024-03-25T19:31:37.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:37.085+0000 7f770ce17640 1 librados: shutdown 2024-03-25T19:31:37.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:43.905 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:44.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.004+0000 7f46dfcfb640 10 monclient: get_monmap_and_config 2024-03-25T19:31:44.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.004+0000 7f46dfcfb640 10 monclient: build_initial_monmap 2024-03-25T19:31:44.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.004+0000 7f46dfcfb640 10 monclient: monmap: 2024-03-25T19:31:44.004 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:44.004 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:44.004 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:31:44.005455+0000 2024-03-25T19:31:44.005 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:31:44.005455+0000 2024-03-25T19:31:44.005 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:44.005 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:44.005 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:44.005 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:31:44.005 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:31:44.005 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:44.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.004+0000 7f46dfcfb640 1 Processor -- start 2024-03-25T19:31:44.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.005+0000 7f46dfcfb640 1 -- start start 2024-03-25T19:31:44.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.005+0000 7f46dfcfb640 10 monclient: init 2024-03-25T19:31:44.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.005+0000 7f46dfcfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:44.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.005+0000 7f46dfcfb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:44.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.005+0000 7f46dfcfb640 10 monclient(hunting): picked mon.noname-a con 0x7f46d80a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:44.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.005+0000 7f46dfcfb640 10 monclient(hunting): picked mon.noname-b con 0x7f46d813f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:44.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.005+0000 7f46dfcfb640 10 monclient(hunting): picked mon.noname-c con 0x7f46d81430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:44.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.005+0000 7f46dfcfb640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d8142520 con 0x7f46d80a98f0 2024-03-25T19:31:44.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.005+0000 7f46dfcfb640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d80b28a0 con 0x7f46d81430c0 2024-03-25T19:31:44.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.005+0000 7f46dfcfb640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d80b2a30 con 0x7f46d813f710 2024-03-25T19:31:44.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.005+0000 7f46dfcfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:44.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.005+0000 7f46dfcfb640 10 monclient(hunting): authenticate will time out at 6157.275750s 2024-03-25T19:31:44.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.011+0000 7f46de298640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f46d81430c0 0x7f46d814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:50380/0 (socket says 172.21.15.19:50380) 2024-03-25T19:31:44.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.011+0000 7f46de298640 1 -- 172.21.15.19:0/614222971 learned_addr learned my addr 172.21.15.19:0/614222971 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:44.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.022+0000 7f46dca95640 1 -- 172.21.15.19:0/614222971 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928544184 0 0) 0x7f46d8142520 con 0x7f46d80a98f0 2024-03-25T19:31:44.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.022+0000 7f46dca95640 10 monclient(hunting): _negotiate 2024-03-25T19:31:44.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.022+0000 7f46dca95640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:44.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.022+0000 7f46dca95640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:44.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.022+0000 7f46dca95640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:31:44.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.022+0000 7f46dca95640 1 -- 172.21.15.19:0/614222971 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c4003520 con 0x7f46d80a98f0 2024-03-25T19:31:44.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.024+0000 7f46dca95640 1 -- 172.21.15.19:0/614222971 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583293740 0 0) 0x7f46d80b28a0 con 0x7f46d81430c0 2024-03-25T19:31:44.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.024+0000 7f46dca95640 10 monclient(hunting): _negotiate 2024-03-25T19:31:44.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.024+0000 7f46dca95640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:44.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.024+0000 7f46dca95640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:44.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.024+0000 7f46dca95640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:31:44.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.024+0000 7f46dca95640 1 -- 172.21.15.19:0/614222971 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d8142520 con 0x7f46d81430c0 2024-03-25T19:31:44.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 1 -- 172.21.15.19:0/614222971 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2069087114 0 0) 0x7f46d8142520 con 0x7f46d81430c0 2024-03-25T19:31:44.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 1 -- 172.21.15.19:0/614222971 >> v1:172.21.15.19:6790/0 conn(0x7f46d813f710 legacy=0x7f46d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:44.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 1 -- 172.21.15.19:0/614222971 >> v1:172.21.15.19:6789/0 conn(0x7f46d80a98f0 legacy=0x7f46d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 10 monclient: _finish_hunting 0 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 1 monclient: found mon.noname-c 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 1 -- 172.21.15.19:0/614222971 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d80b44c0 con 0x7f46d81430c0 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 10 monclient: _finish_auth 0 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 10 monclient: _check_auth_tickets 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dfcfb640 5 monclient: authenticate success, global_id 4345 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 1 -- 172.21.15.19:0/614222971 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f46d00019f0 con 0x7f46d81430c0 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dca95640 10 monclient: dump: 2024-03-25T19:31:44.044 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:44.045 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:44.045 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:44.045 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:44.045 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:44.045 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:44.045 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:44.045 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:44.045 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:44.045 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:44.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.043+0000 7f46dfcfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:44.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.059+0000 7f46dca95640 1 -- 172.21.15.19:0/614222971 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46d0001c10 con 0x7f46d81430c0 2024-03-25T19:31:44.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.059+0000 7f46dca95640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:44.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.059+0000 7f46dca95640 1 -- 172.21.15.19:0/614222971 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f46d0001f20 con 0x7f46d81430c0 2024-03-25T19:31:44.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.059+0000 7f46dca95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:44.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.059+0000 7f46dca95640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:44.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.059+0000 7f46dca95640 10 monclient: dump: 2024-03-25T19:31:44.059 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:44.059 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:44.059 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:44.059 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:44.060 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:44.060 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:44.060 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:44.060 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:44.060 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:44.060 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:44.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.059+0000 7f46dfcfb640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:44.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.059+0000 7f46dfcfb640 4 set_mon_vals no callback set 2024-03-25T19:31:44.062 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.060+0000 7f46dfcfb640 10 monclient: shutdown 2024-03-25T19:31:44.062 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.060+0000 7f46dfcfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:44.062 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.060+0000 7f46dfcfb640 1 -- 172.21.15.19:0/614222971 >> v1:172.21.15.188:6789/0 conn(0x7f46d81430c0 legacy=0x7f46d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:44.062 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.060+0000 7f46dfcfb640 1 -- 172.21.15.19:0/614222971 shutdown_connections 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.060+0000 7f46dfcfb640 1 -- 172.21.15.19:0/614222971 >> 172.21.15.19:0/614222971 conn(0x7f46d80b7ea0 msgr2=0x7f46d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.060+0000 7f46dfcfb640 1 -- 172.21.15.19:0/614222971 shutdown_connections 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.060+0000 7f46dfcfb640 1 -- 172.21.15.19:0/614222971 wait complete. 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.061+0000 7f46dfcfb640 10 monclient: build_initial_monmap 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.061+0000 7f46dfcfb640 10 monclient: monmap: 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:44.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.061+0000 7f46dfcfb640 1 librados: starting msgr at 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.061+0000 7f46dfcfb640 1 librados: starting objecter 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.061+0000 7f46dfcfb640 1 Processor -- start 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.061+0000 7f46dfcfb640 1 -- start start 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.061+0000 7f46dfcfb640 1 librados: setting wanted keys 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.061+0000 7f46dfcfb640 1 librados: calling monclient init 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.061+0000 7f46dfcfb640 10 monclient: init 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.062+0000 7f46dfcfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.062+0000 7f46dfcfb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.062+0000 7f46dfcfb640 10 monclient(hunting): picked mon.noname-c con 0x7f46d80a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.062+0000 7f46dfcfb640 10 monclient(hunting): picked mon.noname-b con 0x7f46d813f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.062+0000 7f46dfcfb640 10 monclient(hunting): picked mon.noname-a con 0x7f46d81430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.062+0000 7f46dfcfb640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d80ad410 con 0x7f46d81430c0 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.062+0000 7f46dfcfb640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d81d7d80 con 0x7f46d813f710 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.062+0000 7f46dfcfb640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d81d8360 con 0x7f46d80a98f0 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.062+0000 7f46dfcfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:44.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.062+0000 7f46dfcfb640 10 monclient(hunting): authenticate will time out at 6157.332357s 2024-03-25T19:31:44.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.072+0000 7f46dd296640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f46d813f710 0x7f46d80acd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:50430/0 (socket says 172.21.15.19:50430) 2024-03-25T19:31:44.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.072+0000 7f46dd296640 1 -- 172.21.15.19:0/266328649 learned_addr learned my addr 172.21.15.19:0/266328649 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:44.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.073+0000 7f46ce7fc640 10 client.?.objecter ms_handle_connect 0x7f46d813f710 2024-03-25T19:31:44.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.073+0000 7f46ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:44.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.081+0000 7f46ce7fc640 10 client.?.objecter ms_handle_connect 0x7f46d80a98f0 2024-03-25T19:31:44.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.081+0000 7f46ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:44.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.084+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383824017 0 0) 0x7f46d81d7d80 con 0x7f46d813f710 2024-03-25T19:31:44.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.084+0000 7f46ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:44.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.084+0000 7f46ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:44.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.084+0000 7f46ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:44.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.084+0000 7f46ce7fc640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:31:44.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.084+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46bc004eb0 con 0x7f46d813f710 2024-03-25T19:31:44.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.085+0000 7f46ce7fc640 10 client.?.objecter ms_handle_connect 0x7f46d81430c0 2024-03-25T19:31:44.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.085+0000 7f46ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:44.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.102+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139936564 0 0) 0x7f46d81d8360 con 0x7f46d80a98f0 2024-03-25T19:31:44.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.102+0000 7f46ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:44.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.102+0000 7f46ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:44.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.102+0000 7f46ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:44.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.102+0000 7f46ce7fc640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:31:44.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.102+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d81d7d80 con 0x7f46d80a98f0 2024-03-25T19:31:44.104 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.104+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901579717 0 0) 0x7f46d80ad410 con 0x7f46d81430c0 2024-03-25T19:31:44.104 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.104+0000 7f46ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:44.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.104+0000 7f46ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:44.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.104+0000 7f46ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:44.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.104+0000 7f46ce7fc640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:31:44.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.104+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d81d8360 con 0x7f46d81430c0 2024-03-25T19:31:44.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.115+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 976583337 0 0) 0x7f46bc004eb0 con 0x7f46d813f710 2024-03-25T19:31:44.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.116+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46d80ad410 con 0x7f46d813f710 2024-03-25T19:31:44.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.116+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f46c8001a30 con 0x7f46d813f710 2024-03-25T19:31:44.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.116+0000 7f46ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:44.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.116+0000 7f46ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:44.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.116+0000 7f46ce7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:44.116 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:44.116 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:44.116 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:44.116 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:44.116 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:44.117 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:44.117 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:44.117 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:44.117 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:44.117 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:44.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.116+0000 7f46ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:44.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.122+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2211131759 0 0) 0x7f46d81d7d80 con 0x7f46d80a98f0 2024-03-25T19:31:44.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.122+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46bc004eb0 con 0x7f46d80a98f0 2024-03-25T19:31:44.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.122+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f46d4001950 con 0x7f46d80a98f0 2024-03-25T19:31:44.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.122+0000 7f46ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:44.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.122+0000 7f46ce7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:44.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.122+0000 7f46ce7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:44.122 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:44.122 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:44.122 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:44.122 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:44.122 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:44.123 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:44.123 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:44.123 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:44.123 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:44.123 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:44.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2612353150 0 0) 0x7f46d80ad410 con 0x7f46d813f710 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 >> v1:172.21.15.19:6790/0 conn(0x7f46d80a98f0 legacy=0x7f46d80b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 >> v1:172.21.15.19:6789/0 conn(0x7f46d81430c0 legacy=0x7f46d81d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46ce7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46ce7fc640 1 monclient: found mon.b 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46ce7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d81d84f0 con 0x7f46d813f710 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46ce7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46ce7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46dfcfb640 5 monclient: authenticate success, global_id 4348 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46dfcfb640 10 monclient: _renew_subs 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46dfcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46dfcfb640 1 -- 172.21.15.19:0/266328649 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d814e560 con 0x7f46d813f710 2024-03-25T19:31:44.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46dfcfb640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:44.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46dfcfb640 10 monclient: _renew_subs 2024-03-25T19:31:44.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46dfcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:44.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46dfcfb640 1 -- 172.21.15.19:0/266328649 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d81d41e0 con 0x7f46d813f710 2024-03-25T19:31:44.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.140+0000 7f46dfcfb640 1 librados: init done 2024-03-25T19:31:44.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.142+0000 7f46dfcfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:44.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.142+0000 7f46dfcfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:44.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.142+0000 7f46dfcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:44.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.142+0000 7f46dfcfb640 1 -- 172.21.15.19:0/266328649 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d80b3ef0 con 0x7f46d813f710 2024-03-25T19:31:44.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.428+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46c8001c70 con 0x7f46d813f710 2024-03-25T19:31:44.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.428+0000 7f46ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:44.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.428+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f46c8003a60 con 0x7f46d813f710 2024-03-25T19:31:44.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.428+0000 7f46ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:44.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.428+0000 7f46ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:44.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.428+0000 7f46ce7fc640 10 monclient: dump: 2024-03-25T19:31:44.428 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:44.429 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:44.429 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:44.429 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:44.429 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:44.429 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:44.429 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:44.429 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:44.429 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:44.429 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:44.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.428+0000 7f46cffff640 4 set_mon_vals no callback set 2024-03-25T19:31:44.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.430+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f46c8003c80 con 0x7f46d813f710 2024-03-25T19:31:44.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.430+0000 7f46dda97640 10 monclient: get_auth_request con 0x7f46bc06df70 auth_method 0 2024-03-25T19:31:44.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.431+0000 7f46ce7fc640 10 client.4348.objecter ms_handle_connect 0x7f46bc06df70 2024-03-25T19:31:44.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.436+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f46c80548a0 con 0x7f46d813f710 2024-03-25T19:31:44.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.436+0000 7f46ce7fc640 10 client.4348.objecter ms_dispatch 0x7f46d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:44.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.436+0000 7f46ce7fc640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:44.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.436+0000 7f46ce7fc640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:44.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.436+0000 7f46ce7fc640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-25T19:31:44.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.437+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46c80503b0 con 0x7f46d813f710 2024-03-25T19:31:44.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.437+0000 7f46ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:44.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.437+0000 7f46ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:44.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.696+0000 7f46dfcfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:44.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.696+0000 7f46dfcfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:44.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.696+0000 7f46dfcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:44.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.696+0000 7f46dfcfb640 1 -- 172.21.15.19:0/266328649 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46d80627b0 con 0x7f46d813f710 2024-03-25T19:31:44.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.723+0000 7f46ce7fc640 1 -- 172.21.15.19:0/266328649 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f46c80522e0 con 0x7f46d813f710 2024-03-25T19:31:44.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.723+0000 7f46ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:44.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.723+0000 7f46ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:44.723 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:31:44.724 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 10 librados: watch_flush enter 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 10 librados: watch_flush exit 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 10 client.4348.objecter successfully canceled tick 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 1 -- 172.21.15.19:0/266328649 >> v1:172.21.15.188:6800/2207506229 conn(0x7f46bc06df70 legacy=0x7f46bc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 10 monclient: shutdown 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 1 -- 172.21.15.19:0/266328649 >> v1:172.21.15.188:6789/0 conn(0x7f46d813f710 legacy=0x7f46d80acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 1 -- 172.21.15.19:0/266328649 shutdown_connections 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 1 -- 172.21.15.19:0/266328649 >> 172.21.15.19:0/266328649 conn(0x7f46d80b7ea0 msgr2=0x7f46d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 1 -- 172.21.15.19:0/266328649 shutdown_connections 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 1 -- 172.21.15.19:0/266328649 wait complete. 2024-03-25T19:31:44.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:44.725+0000 7f46dfcfb640 1 librados: shutdown 2024-03-25T19:31:44.737 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:51.544 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:51.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.661+0000 7efea2e43640 10 monclient: get_monmap_and_config 2024-03-25T19:31:51.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.661+0000 7efea2e43640 10 monclient: build_initial_monmap 2024-03-25T19:31:51.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.661+0000 7efea2e43640 10 monclient: monmap: 2024-03-25T19:31:51.661 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:51.662 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:51.662 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:31:51.662839+0000 2024-03-25T19:31:51.662 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:31:51.662839+0000 2024-03-25T19:31:51.662 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:51.662 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:51.662 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:51.662 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:31:51.662 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:31:51.662 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:51.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.662+0000 7efea2e43640 1 Processor -- start 2024-03-25T19:31:51.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.662+0000 7efea2e43640 1 -- start start 2024-03-25T19:31:51.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.662+0000 7efea2e43640 10 monclient: init 2024-03-25T19:31:51.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.663+0000 7efea2e43640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:51.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.663+0000 7efea2e43640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:51.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.663+0000 7efea2e43640 10 monclient(hunting): picked mon.noname-a con 0x7efe9c0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:51.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.663+0000 7efea2e43640 10 monclient(hunting): picked mon.noname-c con 0x7efe9c13f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:51.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.663+0000 7efea2e43640 10 monclient(hunting): picked mon.noname-b con 0x7efe9c1430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:51.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.663+0000 7efea2e43640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c142520 con 0x7efe9c0a98f0 2024-03-25T19:31:51.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.663+0000 7efea2e43640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0b28a0 con 0x7efe9c13f710 2024-03-25T19:31:51.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.663+0000 7efea2e43640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0b2a30 con 0x7efe9c1430c0 2024-03-25T19:31:51.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.663+0000 7efea2e43640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:51.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.663+0000 7efea2e43640 10 monclient(hunting): authenticate will time out at 6164.933449s 2024-03-25T19:31:51.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.675+0000 7efe93fff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7efe9c13f710 0x7efe9c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:36294/0 (socket says 172.21.15.19:36294) 2024-03-25T19:31:51.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.675+0000 7efe93fff640 1 -- 172.21.15.19:0/1367449971 learned_addr learned my addr 172.21.15.19:0/1367449971 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:51.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.683+0000 7efe937fe640 1 -- 172.21.15.19:0/1367449971 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82488709 0 0) 0x7efe9c0b28a0 con 0x7efe9c13f710 2024-03-25T19:31:51.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.683+0000 7efe937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:51.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.683+0000 7efe937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:51.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.683+0000 7efe937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:51.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.683+0000 7efe937fe640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:31:51.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.683+0000 7efe937fe640 1 -- 172.21.15.19:0/1367449971 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe84003520 con 0x7efe9c13f710 2024-03-25T19:31:51.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.702+0000 7efe937fe640 1 -- 172.21.15.19:0/1367449971 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294351782 0 0) 0x7efe9c142520 con 0x7efe9c0a98f0 2024-03-25T19:31:51.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.702+0000 7efe937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:51.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.702+0000 7efe937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:51.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.702+0000 7efe937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:51.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.702+0000 7efe937fe640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:31:51.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.702+0000 7efe937fe640 1 -- 172.21.15.19:0/1367449971 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c0b28a0 con 0x7efe9c0a98f0 2024-03-25T19:31:51.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 1 -- 172.21.15.19:0/1367449971 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1670576461 0 0) 0x7efe84003520 con 0x7efe9c13f710 2024-03-25T19:31:51.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 1 -- 172.21.15.19:0/1367449971 >> v1:172.21.15.19:6790/0 conn(0x7efe9c1430c0 legacy=0x7efe9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 1 -- 172.21.15.19:0/1367449971 >> v1:172.21.15.19:6789/0 conn(0x7efe9c0a98f0 legacy=0x7efe9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 10 monclient: _finish_hunting 0 2024-03-25T19:31:51.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 1 monclient: found mon.noname-c 2024-03-25T19:31:51.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:51.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:31:51.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 1 -- 172.21.15.19:0/1367449971 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c0b44c0 con 0x7efe9c13f710 2024-03-25T19:31:51.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 10 monclient: _finish_auth 0 2024-03-25T19:31:51.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 10 monclient: _check_auth_tickets 2024-03-25T19:31:51.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:51.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 1 -- 172.21.15.19:0/1367449971 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efe8c001690 con 0x7efe9c13f710 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efe937fe640 10 monclient: dump: 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efea2e43640 5 monclient: authenticate success, global_id 4351 2024-03-25T19:31:51.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.705+0000 7efea2e43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:51.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.720+0000 7efe937fe640 1 -- 172.21.15.19:0/1367449971 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe8c0017f0 con 0x7efe9c13f710 2024-03-25T19:31:51.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.720+0000 7efe937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:51.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.720+0000 7efea2e43640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:51.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.720+0000 7efea2e43640 4 set_mon_vals no callback set 2024-03-25T19:31:51.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.721+0000 7efea2e43640 10 monclient: shutdown 2024-03-25T19:31:51.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.721+0000 7efea2e43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:51.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.721+0000 7efea2e43640 1 -- 172.21.15.19:0/1367449971 >> v1:172.21.15.188:6789/0 conn(0x7efe9c13f710 legacy=0x7efe9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.721+0000 7efea2e43640 1 -- 172.21.15.19:0/1367449971 shutdown_connections 2024-03-25T19:31:51.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.721+0000 7efea2e43640 1 -- 172.21.15.19:0/1367449971 >> 172.21.15.19:0/1367449971 conn(0x7efe9c0b7ea0 msgr2=0x7efe9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:51.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.721+0000 7efea2e43640 1 -- 172.21.15.19:0/1367449971 shutdown_connections 2024-03-25T19:31:51.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.721+0000 7efea2e43640 1 -- 172.21.15.19:0/1367449971 wait complete. 2024-03-25T19:31:51.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.722+0000 7efea2e43640 10 monclient: build_initial_monmap 2024-03-25T19:31:51.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.722+0000 7efea2e43640 10 monclient: monmap: 2024-03-25T19:31:51.723 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:51.723 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:51.723 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:31:51.723 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:31:51.723 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:51.723 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:31:51.723 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:51.723 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:31:51.724 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:31:51.724 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:51.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.723+0000 7efea2e43640 1 librados: starting msgr at 2024-03-25T19:31:51.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.723+0000 7efea2e43640 1 librados: starting objecter 2024-03-25T19:31:51.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.723+0000 7efea2e43640 1 Processor -- start 2024-03-25T19:31:51.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.723+0000 7efea2e43640 1 -- start start 2024-03-25T19:31:51.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.723+0000 7efea2e43640 1 librados: setting wanted keys 2024-03-25T19:31:51.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.723+0000 7efea2e43640 1 librados: calling monclient init 2024-03-25T19:31:51.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.723+0000 7efea2e43640 10 monclient: init 2024-03-25T19:31:51.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.724+0000 7efea2e43640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:51.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.724+0000 7efea2e43640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:51.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.724+0000 7efea2e43640 10 monclient(hunting): picked mon.noname-b con 0x7efe9c0a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:51.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.724+0000 7efea2e43640 10 monclient(hunting): picked mon.noname-c con 0x7efe9c13f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:51.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.724+0000 7efea2e43640 10 monclient(hunting): picked mon.noname-a con 0x7efe9c1430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:51.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.725+0000 7efea2e43640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c139680 con 0x7efe9c1430c0 2024-03-25T19:31:51.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.725+0000 7efea2e43640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c138cf0 con 0x7efe9c0a98f0 2024-03-25T19:31:51.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.725+0000 7efea2e43640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c136620 con 0x7efe9c13f710 2024-03-25T19:31:51.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.725+0000 7efea2e43640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:51.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.725+0000 7efea2e43640 10 monclient(hunting): authenticate will time out at 6164.994965s 2024-03-25T19:31:51.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.731+0000 7efea0bdf640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7efe9c0a98f0 0x7efe9c1371a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:36356/0 (socket says 172.21.15.19:36356) 2024-03-25T19:31:51.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.731+0000 7efea0bdf640 1 -- 172.21.15.19:0/3618906653 learned_addr learned my addr 172.21.15.19:0/3618906653 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:51.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.734+0000 7efe917fa640 10 client.?.objecter ms_handle_connect 0x7efe9c0a98f0 2024-03-25T19:31:51.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.734+0000 7efe917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:51.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.744+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768054814 0 0) 0x7efe9c138cf0 con 0x7efe9c0a98f0 2024-03-25T19:31:51.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.744+0000 7efe917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:31:51.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.744+0000 7efe917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:51.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.744+0000 7efe917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:51.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.744+0000 7efe917fa640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:31:51.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.744+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe88004eb0 con 0x7efe9c0a98f0 2024-03-25T19:31:51.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.745+0000 7efe917fa640 10 client.?.objecter ms_handle_connect 0x7efe9c13f710 2024-03-25T19:31:51.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.745+0000 7efe917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:51.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.754+0000 7efe917fa640 10 client.?.objecter ms_handle_connect 0x7efe9c1430c0 2024-03-25T19:31:51.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.754+0000 7efe917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:51.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.760+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857478476 0 0) 0x7efe9c136620 con 0x7efe9c13f710 2024-03-25T19:31:51.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.760+0000 7efe917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:31:51.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.760+0000 7efe917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:51.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.760+0000 7efe917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:51.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.760+0000 7efe917fa640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:31:51.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.760+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c138cf0 con 0x7efe9c13f710 2024-03-25T19:31:51.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.766+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2673589820 0 0) 0x7efe88004eb0 con 0x7efe9c0a98f0 2024-03-25T19:31:51.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.766+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe9c136620 con 0x7efe9c0a98f0 2024-03-25T19:31:51.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.766+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efe98001d90 con 0x7efe9c0a98f0 2024-03-25T19:31:51.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.766+0000 7efe917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:51.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.766+0000 7efe917fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:51.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.766+0000 7efe917fa640 10 monclient(hunting): dump: 2024-03-25T19:31:51.767 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:51.767 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:51.767 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:51.767 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:51.767 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:51.767 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:51.767 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:51.767 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:51.767 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:51.767 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:51.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.766+0000 7efe917fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:51.769 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.770+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013935571 0 0) 0x7efe9c139680 con 0x7efe9c1430c0 2024-03-25T19:31:51.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.770+0000 7efe917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:31:51.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.770+0000 7efe917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:51.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.770+0000 7efe917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:51.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.770+0000 7efe917fa640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:31:51.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.770+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe88004eb0 con 0x7efe9c1430c0 2024-03-25T19:31:51.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3807153691 0 0) 0x7efe9c136620 con 0x7efe9c0a98f0 2024-03-25T19:31:51.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 >> v1:172.21.15.19:6790/0 conn(0x7efe9c13f710 legacy=0x7efe9c1dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 >> v1:172.21.15.19:6789/0 conn(0x7efe9c1430c0 legacy=0x7efe9c1e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efe917fa640 10 monclient: _finish_hunting 0 2024-03-25T19:31:51.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efe917fa640 1 monclient: found mon.b 2024-03-25T19:31:51.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efe917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:51.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efe917fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:51.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c0b1620 con 0x7efe9c0a98f0 2024-03-25T19:31:51.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efe917fa640 10 monclient: _finish_auth 0 2024-03-25T19:31:51.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efe917fa640 10 monclient: _check_auth_tickets 2024-03-25T19:31:51.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efe917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:51.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efea2e43640 5 monclient: authenticate success, global_id 4354 2024-03-25T19:31:51.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efea2e43640 10 monclient: _renew_subs 2024-03-25T19:31:51.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efea2e43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:51.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efea2e43640 1 -- 172.21.15.19:0/3618906653 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c139110 con 0x7efe9c0a98f0 2024-03-25T19:31:51.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efea2e43640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:51.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efea2e43640 10 monclient: _renew_subs 2024-03-25T19:31:51.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efea2e43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:51.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efea2e43640 1 -- 172.21.15.19:0/3618906653 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c13b770 con 0x7efe9c0a98f0 2024-03-25T19:31:51.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.784+0000 7efea2e43640 1 librados: init done 2024-03-25T19:31:51.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.785+0000 7efea2e43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:51.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.785+0000 7efea2e43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:51.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.785+0000 7efea2e43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:51.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:51.785+0000 7efea2e43640 1 -- 172.21.15.19:0/3618906653 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c0627b0 con 0x7efe9c0a98f0 2024-03-25T19:31:52.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.057+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe98001fd0 con 0x7efe9c0a98f0 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.057+0000 7efe917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.057+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efe98003de0 con 0x7efe9c0a98f0 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.057+0000 7efe917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.057+0000 7efe917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.057+0000 7efe917fa640 10 monclient: dump: 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:52.058 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:52.059 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:52.059 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:52.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.057+0000 7efe927fc640 4 set_mon_vals no callback set 2024-03-25T19:31:52.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.058+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7efe98004000 con 0x7efe9c0a98f0 2024-03-25T19:31:52.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.059+0000 7efe93fff640 10 monclient: get_auth_request con 0x7efe8806de00 auth_method 0 2024-03-25T19:31:52.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.059+0000 7efe917fa640 10 client.4354.objecter ms_handle_connect 0x7efe8806de00 2024-03-25T19:31:52.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.064+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7efe98054c10 con 0x7efe9c0a98f0 2024-03-25T19:31:52.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.064+0000 7efe917fa640 10 client.4354.objecter ms_dispatch 0x7efe9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:52.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.064+0000 7efe917fa640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:52.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.064+0000 7efe917fa640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:52.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.064+0000 7efe917fa640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-25T19:31:52.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.064+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe98050720 con 0x7efe9c0a98f0 2024-03-25T19:31:52.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.064+0000 7efe917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:52.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.064+0000 7efe917fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:52.238 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:14.967 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:52.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.276+0000 7efea2e43640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:52.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.276+0000 7efea2e43640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:52.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.276+0000 7efea2e43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:31:52.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.276+0000 7efea2e43640 1 -- 172.21.15.19:0/3618906653 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe9c069f40 con 0x7efe9c0a98f0 2024-03-25T19:31:52.290 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:15.019 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:52.326 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:15.055 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:52.357 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:15.086 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:52.362 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:15.092 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:52.371 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:03:15.074 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:52.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.375+0000 7efe917fa640 1 -- 172.21.15.19:0/3618906653 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7efe98052650 con 0x7efe9c0a98f0 2024-03-25T19:31:52.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.375+0000 7efe917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:52.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.375+0000 7efe917fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:52.376 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:31:52.376 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:52.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.377+0000 7efea2e43640 10 librados: watch_flush enter 2024-03-25T19:31:52.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.378+0000 7efea2e43640 10 librados: watch_flush exit 2024-03-25T19:31:52.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.378+0000 7efea2e43640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-25T19:31:52.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.378+0000 7efea2e43640 10 client.4354.objecter successfully canceled tick 2024-03-25T19:31:52.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.378+0000 7efea2e43640 1 -- 172.21.15.19:0/3618906653 >> v1:172.21.15.188:6800/2207506229 conn(0x7efe8806de00 legacy=0x7efe880702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:52.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.378+0000 7efea2e43640 10 monclient: shutdown 2024-03-25T19:31:52.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.378+0000 7efea2e43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:52.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.378+0000 7efea2e43640 1 -- 172.21.15.19:0/3618906653 >> v1:172.21.15.188:6789/0 conn(0x7efe9c0a98f0 legacy=0x7efe9c1371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:52.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.378+0000 7efea2e43640 1 -- 172.21.15.19:0/3618906653 shutdown_connections 2024-03-25T19:31:52.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.378+0000 7efea2e43640 1 -- 172.21.15.19:0/3618906653 >> 172.21.15.19:0/3618906653 conn(0x7efe9c0b7ea0 msgr2=0x7efe9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:52.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.378+0000 7efea2e43640 1 -- 172.21.15.19:0/3618906653 shutdown_connections 2024-03-25T19:31:52.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.378+0000 7efea2e43640 1 -- 172.21.15.19:0/3618906653 wait complete. 2024-03-25T19:31:52.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:52.378+0000 7efea2e43640 1 librados: shutdown 2024-03-25T19:31:52.380 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:03:15.081 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:52.391 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:52.402 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:03:15.105 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:52.405 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:15.134 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:59.197 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:59.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.303+0000 7f2f5be1e640 10 monclient: get_monmap_and_config 2024-03-25T19:31:59.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.303+0000 7f2f5be1e640 10 monclient: build_initial_monmap 2024-03-25T19:31:59.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.303+0000 7f2f5be1e640 10 monclient: monmap: 2024-03-25T19:31:59.303 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:59.303 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:59.303 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:31:59.303975+0000 2024-03-25T19:31:59.303 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:31:59.303975+0000 2024-03-25T19:31:59.303 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:59.303 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:59.303 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:59.303 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:31:59.303 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:31:59.303 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:59.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.304+0000 7f2f5be1e640 1 Processor -- start 2024-03-25T19:31:59.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.304+0000 7f2f5be1e640 1 -- start start 2024-03-25T19:31:59.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.304+0000 7f2f5be1e640 10 monclient: init 2024-03-25T19:31:59.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.305+0000 7f2f5be1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:59.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.305+0000 7f2f5be1e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:59.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.305+0000 7f2f5be1e640 10 monclient(hunting): picked mon.noname-a con 0x7f2f540b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:59.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.305+0000 7f2f5be1e640 10 monclient(hunting): picked mon.noname-b con 0x7f2f540b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:59.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.305+0000 7f2f5be1e640 10 monclient(hunting): picked mon.noname-c con 0x7f2f5414be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:59.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.305+0000 7f2f5be1e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540af940 con 0x7f2f540b4d60 2024-03-25T19:31:59.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.305+0000 7f2f5be1e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540b0a70 con 0x7f2f5414be90 2024-03-25T19:31:59.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.305+0000 7f2f5be1e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f540b0d50 con 0x7f2f540b1730 2024-03-25T19:31:59.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.305+0000 7f2f5be1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:59.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.305+0000 7f2f5be1e640 10 monclient(hunting): authenticate will time out at 6172.574052s 2024-03-25T19:31:59.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.336+0000 7f2f5a3bb640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f2f5414be90 0x7f2f5414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:40910/0 (socket says 172.21.15.19:40910) 2024-03-25T19:31:59.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.336+0000 7f2f5a3bb640 1 -- 172.21.15.19:0/3446299185 learned_addr learned my addr 172.21.15.19:0/3446299185 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:59.353 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.354+0000 7f2f58bb8640 1 -- 172.21.15.19:0/3446299185 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546694938 0 0) 0x7f2f540b0d50 con 0x7f2f540b1730 2024-03-25T19:31:59.353 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.354+0000 7f2f58bb8640 10 monclient(hunting): _negotiate 2024-03-25T19:31:59.353 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.354+0000 7f2f58bb8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:59.353 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.354+0000 7f2f58bb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:59.353 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.354+0000 7f2f58bb8640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:31:59.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.354+0000 7f2f58bb8640 1 -- 172.21.15.19:0/3446299185 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f40003520 con 0x7f2f540b1730 2024-03-25T19:31:59.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.355+0000 7f2f58bb8640 1 -- 172.21.15.19:0/3446299185 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766316758 0 0) 0x7f2f540b0a70 con 0x7f2f5414be90 2024-03-25T19:31:59.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.355+0000 7f2f58bb8640 10 monclient(hunting): _negotiate 2024-03-25T19:31:59.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.355+0000 7f2f58bb8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:59.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.355+0000 7f2f58bb8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:59.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.355+0000 7f2f58bb8640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:31:59.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.355+0000 7f2f58bb8640 1 -- 172.21.15.19:0/3446299185 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f540b0d50 con 0x7f2f5414be90 2024-03-25T19:31:59.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 1 -- 172.21.15.19:0/3446299185 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 661295723 0 0) 0x7f2f40003520 con 0x7f2f540b1730 2024-03-25T19:31:59.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 1 -- 172.21.15.19:0/3446299185 >> v1:172.21.15.188:6789/0 conn(0x7f2f5414be90 legacy=0x7f2f5414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:59.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 1 -- 172.21.15.19:0/3446299185 >> v1:172.21.15.19:6789/0 conn(0x7f2f540b4d60 legacy=0x7f2f5413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:59.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 10 monclient: _finish_hunting 0 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 1 monclient: found mon.noname-b 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 1 -- 172.21.15.19:0/3446299185 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f540b1030 con 0x7f2f540b1730 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 10 monclient: _finish_auth 0 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 10 monclient: _check_auth_tickets 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 1 -- 172.21.15.19:0/3446299185 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2f44001690 con 0x7f2f540b1730 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f5be1e640 5 monclient: authenticate success, global_id 4334 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f5be1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f58bb8640 10 monclient: dump: 2024-03-25T19:31:59.368 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:59.369 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:59.369 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:59.369 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:59.369 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:59.369 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:59.369 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:59.369 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:59.369 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:59.369 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:59.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.368+0000 7f2f5be1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:59.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.389+0000 7f2f58bb8640 1 -- 172.21.15.19:0/3446299185 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f440017f0 con 0x7f2f540b1730 2024-03-25T19:31:59.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.389+0000 7f2f58bb8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:59.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.389+0000 7f2f58bb8640 1 -- 172.21.15.19:0/3446299185 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2f44001c00 con 0x7f2f540b1730 2024-03-25T19:31:59.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.389+0000 7f2f5be1e640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.389+0000 7f2f5be1e640 4 set_mon_vals no callback set 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.389+0000 7f2f58bb8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.389+0000 7f2f58bb8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.389+0000 7f2f58bb8640 10 monclient: dump: 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:59.389 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:59.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.390+0000 7f2f5be1e640 10 monclient: shutdown 2024-03-25T19:31:59.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.390+0000 7f2f5be1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:59.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.390+0000 7f2f5be1e640 1 -- 172.21.15.19:0/3446299185 >> v1:172.21.15.19:6790/0 conn(0x7f2f540b1730 legacy=0x7f2f540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:59.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.390+0000 7f2f5be1e640 1 -- 172.21.15.19:0/3446299185 shutdown_connections 2024-03-25T19:31:59.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.390+0000 7f2f5be1e640 1 -- 172.21.15.19:0/3446299185 >> 172.21.15.19:0/3446299185 conn(0x7f2f540617a0 msgr2=0x7f2f54061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:59.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.390+0000 7f2f5be1e640 1 -- 172.21.15.19:0/3446299185 shutdown_connections 2024-03-25T19:31:59.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.390+0000 7f2f5be1e640 1 -- 172.21.15.19:0/3446299185 wait complete. 2024-03-25T19:31:59.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.391+0000 7f2f5be1e640 10 monclient: build_initial_monmap 2024-03-25T19:31:59.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.391+0000 7f2f5be1e640 10 monclient: monmap: 2024-03-25T19:31:59.391 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:31:59.391 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.392+0000 7f2f5be1e640 1 librados: starting msgr at 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.392+0000 7f2f5be1e640 1 librados: starting objecter 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.392+0000 7f2f5be1e640 1 Processor -- start 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.392+0000 7f2f5be1e640 1 -- start start 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.392+0000 7f2f5be1e640 1 librados: setting wanted keys 2024-03-25T19:31:59.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.392+0000 7f2f5be1e640 1 librados: calling monclient init 2024-03-25T19:31:59.393 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.392+0000 7f2f5be1e640 10 monclient: init 2024-03-25T19:31:59.393 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.393+0000 7f2f5be1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:59.393 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.393+0000 7f2f5be1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:59.393 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.393+0000 7f2f5be1e640 10 monclient(hunting): picked mon.noname-b con 0x7f2f540b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:31:59.393 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.393+0000 7f2f5be1e640 10 monclient(hunting): picked mon.noname-a con 0x7f2f540b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:31:59.394 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.393+0000 7f2f5be1e640 10 monclient(hunting): picked mon.noname-c con 0x7f2f5414be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:31:59.394 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.393+0000 7f2f5be1e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f54143900 con 0x7f2f540b4d60 2024-03-25T19:31:59.394 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.393+0000 7f2f5be1e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5413a550 con 0x7f2f540b1730 2024-03-25T19:31:59.394 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.393+0000 7f2f5be1e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5413bc40 con 0x7f2f5414be90 2024-03-25T19:31:59.394 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.393+0000 7f2f5be1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:59.394 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.393+0000 7f2f5be1e640 10 monclient(hunting): authenticate will time out at 6172.662521s 2024-03-25T19:31:59.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.401+0000 7f2f59bba640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f2f540b1730 0x7f2f54139e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:40954/0 (socket says 172.21.15.19:40954) 2024-03-25T19:31:59.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.401+0000 7f2f59bba640 1 -- 172.21.15.19:0/1564067298 learned_addr learned my addr 172.21.15.19:0/1564067298 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:31:59.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.403+0000 7f2f4a7fc640 10 client.?.objecter ms_handle_connect 0x7f2f540b1730 2024-03-25T19:31:59.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.403+0000 7f2f4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:59.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.409+0000 7f2f4a7fc640 10 client.?.objecter ms_handle_connect 0x7f2f5414be90 2024-03-25T19:31:59.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.409+0000 7f2f4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:59.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.410+0000 7f2f4a7fc640 10 client.?.objecter ms_handle_connect 0x7f2f540b4d60 2024-03-25T19:31:59.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.410+0000 7f2f4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:59.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.417+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935333582 0 0) 0x7f2f5413a550 con 0x7f2f540b1730 2024-03-25T19:31:59.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.417+0000 7f2f4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:59.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.417+0000 7f2f4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:59.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.417+0000 7f2f4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:59.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.417+0000 7f2f4a7fc640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:31:59.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.417+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f38004eb0 con 0x7f2f540b1730 2024-03-25T19:31:59.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.420+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748993995 0 0) 0x7f2f5413bc40 con 0x7f2f5414be90 2024-03-25T19:31:59.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.420+0000 7f2f4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:59.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.420+0000 7f2f4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:59.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.420+0000 7f2f4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:59.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.420+0000 7f2f4a7fc640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:31:59.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.420+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5413a550 con 0x7f2f5414be90 2024-03-25T19:31:59.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.421+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720282595 0 0) 0x7f2f54143900 con 0x7f2f540b4d60 2024-03-25T19:31:59.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.421+0000 7f2f4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:59.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.421+0000 7f2f4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:59.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.421+0000 7f2f4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:59.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.421+0000 7f2f4a7fc640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:31:59.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.421+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5413bc40 con 0x7f2f540b4d60 2024-03-25T19:31:59.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.429+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1487446414 0 0) 0x7f2f38004eb0 con 0x7f2f540b1730 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.429+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f54143900 con 0x7f2f540b1730 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.429+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2f50001a30 con 0x7f2f540b1730 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.429+0000 7f2f4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.429+0000 7f2f4a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.429+0000 7f2f4a7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:59.428 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:59.429 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:59.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.429+0000 7f2f4a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:59.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.431+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2755705192 0 0) 0x7f2f5413a550 con 0x7f2f5414be90 2024-03-25T19:31:59.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.431+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f38004eb0 con 0x7f2f5414be90 2024-03-25T19:31:59.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.431+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2f4c001b50 con 0x7f2f5414be90 2024-03-25T19:31:59.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.431+0000 7f2f4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:59.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.431+0000 7f2f4a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:59.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.431+0000 7f2f4a7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:59.430 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:59.430 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:59.430 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:59.430 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:59.431 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:59.431 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:59.431 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:59.431 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:59.431 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:59.431 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:59.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.434+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4160458249 0 0) 0x7f2f5413bc40 con 0x7f2f540b4d60 2024-03-25T19:31:59.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.434+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f5413a550 con 0x7f2f540b4d60 2024-03-25T19:31:59.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.434+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2f44001b00 con 0x7f2f540b4d60 2024-03-25T19:31:59.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.434+0000 7f2f4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:59.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.434+0000 7f2f4a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:59.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.434+0000 7f2f4a7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:59.433 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:59.434 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:59.434 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:59.434 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:59.434 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:59.434 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:59.434 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:59.434 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:59.434 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:59.434 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:59.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4149211761 0 0) 0x7f2f38004eb0 con 0x7f2f5414be90 2024-03-25T19:31:59.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 >> v1:172.21.15.188:6789/0 conn(0x7f2f540b1730 legacy=0x7f2f54139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:59.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 >> v1:172.21.15.19:6789/0 conn(0x7f2f540b4d60 legacy=0x7f2f5412fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:59.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:59.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f4a7fc640 1 monclient: found mon.c 2024-03-25T19:31:59.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:59.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f4a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:31:59.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5413bdd0 con 0x7f2f5414be90 2024-03-25T19:31:59.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f4a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:59.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:59.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:59.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f5be1e640 5 monclient: authenticate success, global_id 4337 2024-03-25T19:31:59.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.440+0000 7f2f5be1e640 10 monclient: _renew_subs 2024-03-25T19:31:59.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.441+0000 7f2f5be1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:31:59.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.441+0000 7f2f5be1e640 1 -- 172.21.15.19:0/1564067298 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f541391c0 con 0x7f2f5414be90 2024-03-25T19:31:59.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.441+0000 7f2f5be1e640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:59.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.441+0000 7f2f5be1e640 10 monclient: _renew_subs 2024-03-25T19:31:59.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.441+0000 7f2f5be1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:31:59.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.441+0000 7f2f5be1e640 1 -- 172.21.15.19:0/1564067298 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f54130670 con 0x7f2f5414be90 2024-03-25T19:31:59.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.441+0000 7f2f5be1e640 1 librados: init done 2024-03-25T19:31:59.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.442+0000 7f2f5be1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:59.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.442+0000 7f2f5be1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:59.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.442+0000 7f2f5be1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:31:59.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.442+0000 7f2f5be1e640 1 -- 172.21.15.19:0/1564067298 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f541d7a80 con 0x7f2f5414be90 2024-03-25T19:31:59.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.445+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f4c002000 con 0x7f2f5414be90 2024-03-25T19:31:59.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.445+0000 7f2f4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:59.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.445+0000 7f2f4bfff640 4 set_mon_vals no callback set 2024-03-25T19:31:59.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.753+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2f4c004d00 con 0x7f2f5414be90 2024-03-25T19:31:59.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.753+0000 7f2f4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:59.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.753+0000 7f2f4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:59.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.753+0000 7f2f4a7fc640 10 monclient: dump: 2024-03-25T19:31:59.752 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:31:59.753 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:31:59.753 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:59.753 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:31:59.753 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:59.753 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:31:59.753 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:31:59.753 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:31:59.753 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:31:59.753 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:31:59.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.754+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f2f4c01acd0 con 0x7f2f5414be90 2024-03-25T19:31:59.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.754+0000 7f2f59bba640 10 monclient: get_auth_request con 0x7f2f3806e180 auth_method 0 2024-03-25T19:31:59.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.755+0000 7f2f4a7fc640 10 client.4337.objecter ms_handle_connect 0x7f2f3806e180 2024-03-25T19:31:59.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.761+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f2f4c0548f0 con 0x7f2f5414be90 2024-03-25T19:31:59.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.761+0000 7f2f4a7fc640 10 client.4337.objecter ms_dispatch 0x7f2f54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:59.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.761+0000 7f2f4a7fc640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:59.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.761+0000 7f2f4a7fc640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:59.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.761+0000 7f2f4a7fc640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-25T19:31:59.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.761+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f4c0b18b0 con 0x7f2f5414be90 2024-03-25T19:31:59.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.761+0000 7f2f4a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:59.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:31:59.761+0000 7f2f4a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:00.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.012+0000 7f2f5be1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:00.012 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.012+0000 7f2f5be1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:00.012 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.012+0000 7f2f5be1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:32:00.012 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.012+0000 7f2f5be1e640 1 -- 172.21.15.19:0/1564067298 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f540627b0 con 0x7f2f5414be90 2024-03-25T19:32:00.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.036+0000 7f2f4a7fc640 1 -- 172.21.15.19:0/1564067298 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f2f4c050230 con 0x7f2f5414be90 2024-03-25T19:32:00.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.036+0000 7f2f4a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:00.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.036+0000 7f2f4a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:00.035 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:32:00.036 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:00.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 10 librados: watch_flush enter 2024-03-25T19:32:00.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 10 librados: watch_flush exit 2024-03-25T19:32:00.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-25T19:32:00.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 10 client.4337.objecter successfully canceled tick 2024-03-25T19:32:00.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 1 -- 172.21.15.19:0/1564067298 >> v1:172.21.15.188:6800/2207506229 conn(0x7f2f3806e180 legacy=0x7f2f38070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:00.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 10 monclient: shutdown 2024-03-25T19:32:00.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:00.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 1 -- 172.21.15.19:0/1564067298 >> v1:172.21.15.19:6790/0 conn(0x7f2f5414be90 legacy=0x7f2f541d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:00.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 1 -- 172.21.15.19:0/1564067298 shutdown_connections 2024-03-25T19:32:00.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 1 -- 172.21.15.19:0/1564067298 >> 172.21.15.19:0/1564067298 conn(0x7f2f540617a0 msgr2=0x7f2f54061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:00.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 1 -- 172.21.15.19:0/1564067298 shutdown_connections 2024-03-25T19:32:00.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 1 -- 172.21.15.19:0/1564067298 wait complete. 2024-03-25T19:32:00.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:00.038+0000 7f2f5be1e640 1 librados: shutdown 2024-03-25T19:32:00.050 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:06.856 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:06.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.972+0000 7f7265a25640 10 monclient: get_monmap_and_config 2024-03-25T19:32:06.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.972+0000 7f7265a25640 10 monclient: build_initial_monmap 2024-03-25T19:32:06.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.972+0000 7f7265a25640 10 monclient: monmap: 2024-03-25T19:32:06.971 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:06.971 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:06.971 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:32:06.972452+0000 2024-03-25T19:32:06.972 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:32:06.972452+0000 2024-03-25T19:32:06.972 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:06.972 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:06.972 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:06.972 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:32:06.972 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:32:06.972 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:06.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 1 Processor -- start 2024-03-25T19:32:06.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 1 -- start start 2024-03-25T19:32:06.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 10 monclient: init 2024-03-25T19:32:06.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:06.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:06.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 10 monclient(hunting): picked mon.noname-c con 0x7f72600a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:06.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 10 monclient(hunting): picked mon.noname-b con 0x7f726013f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:06.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 10 monclient(hunting): picked mon.noname-a con 0x7f72601430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:06.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7260142520 con 0x7f72601430c0 2024-03-25T19:32:06.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72600b28a0 con 0x7f72600a98f0 2024-03-25T19:32:06.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72600b2a30 con 0x7f726013f710 2024-03-25T19:32:06.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:06.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.973+0000 7f7265a25640 10 monclient(hunting): authenticate will time out at 6180.242801s 2024-03-25T19:32:06.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.974+0000 7f725effd640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f72600a98f0 0x7f72600a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:36358/0 (socket says 172.21.15.19:36358) 2024-03-25T19:32:06.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:06.974+0000 7f725effd640 1 -- 172.21.15.19:0/1566507194 learned_addr learned my addr 172.21.15.19:0/1566507194 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:07.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.012+0000 7f725e7fc640 1 -- 172.21.15.19:0/1566507194 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201835750 0 0) 0x7f7260142520 con 0x7f72601430c0 2024-03-25T19:32:07.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.012+0000 7f725e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:07.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.012+0000 7f725e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:07.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.012+0000 7f725e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:07.012 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.012+0000 7f725e7fc640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:32:07.012 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.012+0000 7f725e7fc640 1 -- 172.21.15.19:0/1566507194 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f724c003520 con 0x7f72601430c0 2024-03-25T19:32:07.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 1 -- 172.21.15.19:0/1566507194 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3859058527 0 0) 0x7f724c003520 con 0x7f72601430c0 2024-03-25T19:32:07.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 1 -- 172.21.15.19:0/1566507194 >> v1:172.21.15.19:6790/0 conn(0x7f726013f710 legacy=0x7f72601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:07.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 1 -- 172.21.15.19:0/1566507194 >> v1:172.21.15.188:6789/0 conn(0x7f72600a98f0 legacy=0x7f72600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:07.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:32:07.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 1 monclient: found mon.noname-a 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 1 -- 172.21.15.19:0/1566507194 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72600b44c0 con 0x7f72601430c0 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 1 -- 172.21.15.19:0/1566507194 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7258001a10 con 0x7f72601430c0 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f7265a25640 5 monclient: authenticate success, global_id 4368 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f7265a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.098+0000 7f725e7fc640 10 monclient: dump: 2024-03-25T19:32:07.099 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:07.100 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:07.100 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:07.100 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:07.100 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:07.100 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:07.100 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:07.100 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:07.100 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:07.100 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:07.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.099+0000 7f7265a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:07.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.133+0000 7f725e7fc640 1 -- 172.21.15.19:0/1566507194 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7258001b70 con 0x7f72601430c0 2024-03-25T19:32:07.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.133+0000 7f725e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:07.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.133+0000 7f725e7fc640 1 -- 172.21.15.19:0/1566507194 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7258001f80 con 0x7f72601430c0 2024-03-25T19:32:07.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.133+0000 7f7265a25640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:07.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.133+0000 7f7265a25640 4 set_mon_vals no callback set 2024-03-25T19:32:07.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.133+0000 7f725e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:07.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.133+0000 7f725e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:07.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.133+0000 7f725e7fc640 10 monclient: dump: 2024-03-25T19:32:07.133 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:07.133 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:07.133 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:07.133 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:07.133 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:07.133 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:07.133 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:07.133 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:07.133 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:07.133 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:07.136 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.134+0000 7f7265a25640 10 monclient: shutdown 2024-03-25T19:32:07.136 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.134+0000 7f7265a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:07.136 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.134+0000 7f7265a25640 1 -- 172.21.15.19:0/1566507194 >> v1:172.21.15.19:6789/0 conn(0x7f72601430c0 legacy=0x7f726014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:07.136 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.134+0000 7f7265a25640 1 -- 172.21.15.19:0/1566507194 shutdown_connections 2024-03-25T19:32:07.136 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.134+0000 7f7265a25640 1 -- 172.21.15.19:0/1566507194 >> 172.21.15.19:0/1566507194 conn(0x7f72600b7ea0 msgr2=0x7f72600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:07.136 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.134+0000 7f7265a25640 1 -- 172.21.15.19:0/1566507194 shutdown_connections 2024-03-25T19:32:07.136 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.134+0000 7f7265a25640 1 -- 172.21.15.19:0/1566507194 wait complete. 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.135+0000 7f7265a25640 10 monclient: build_initial_monmap 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.135+0000 7f7265a25640 10 monclient: monmap: 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:07.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.135+0000 7f7265a25640 1 librados: starting msgr at 2024-03-25T19:32:07.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.135+0000 7f7265a25640 1 librados: starting objecter 2024-03-25T19:32:07.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.135+0000 7f7265a25640 1 Processor -- start 2024-03-25T19:32:07.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.135+0000 7f7265a25640 1 -- start start 2024-03-25T19:32:07.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.135+0000 7f7265a25640 1 librados: setting wanted keys 2024-03-25T19:32:07.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.135+0000 7f7265a25640 1 librados: calling monclient init 2024-03-25T19:32:07.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.135+0000 7f7265a25640 10 monclient: init 2024-03-25T19:32:07.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.136+0000 7f7265a25640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:07.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.136+0000 7f7265a25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:07.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.136+0000 7f7265a25640 10 monclient(hunting): picked mon.noname-a con 0x7f72600a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:07.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.136+0000 7f7265a25640 10 monclient(hunting): picked mon.noname-b con 0x7f726013f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:07.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.136+0000 7f7265a25640 10 monclient(hunting): picked mon.noname-c con 0x7f72601430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:07.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.136+0000 7f7265a25640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7260138b60 con 0x7f72600a98f0 2024-03-25T19:32:07.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.136+0000 7f7265a25640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7260137e40 con 0x7f726013f710 2024-03-25T19:32:07.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.136+0000 7f7265a25640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72601d82f0 con 0x7f72601430c0 2024-03-25T19:32:07.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.136+0000 7f7265a25640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:07.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.136+0000 7f7265a25640 10 monclient(hunting): authenticate will time out at 6180.405484s 2024-03-25T19:32:07.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.142+0000 7f725effd640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f72600a98f0 0x7f7260137730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:40896/0 (socket says 172.21.15.19:40896) 2024-03-25T19:32:07.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.142+0000 7f725effd640 1 -- 172.21.15.19:0/4159816712 learned_addr learned my addr 172.21.15.19:0/4159816712 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:07.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.143+0000 7f725cff9640 10 client.?.objecter ms_handle_connect 0x7f72600a98f0 2024-03-25T19:32:07.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.143+0000 7f725cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:07.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.144+0000 7f725cff9640 10 client.?.objecter ms_handle_connect 0x7f726013f710 2024-03-25T19:32:07.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.144+0000 7f725cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:07.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.159+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534118774 0 0) 0x7f7260138b60 con 0x7f72600a98f0 2024-03-25T19:32:07.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.159+0000 7f725cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:07.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.159+0000 7f725cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:07.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.159+0000 7f725cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:07.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.159+0000 7f725cff9640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:32:07.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.159+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7240004eb0 con 0x7f72600a98f0 2024-03-25T19:32:07.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.168+0000 7f725cff9640 10 client.?.objecter ms_handle_connect 0x7f72601430c0 2024-03-25T19:32:07.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.168+0000 7f725cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:07.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.176+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1173277197 0 0) 0x7f7240004eb0 con 0x7f72600a98f0 2024-03-25T19:32:07.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.176+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7260138b60 con 0x7f72600a98f0 2024-03-25T19:32:07.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.176+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f72500018b0 con 0x7f72600a98f0 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.176+0000 7f725cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.176+0000 7f725cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.176+0000 7f725cff9640 10 monclient(hunting): dump: 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:07.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.176+0000 7f725cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:07.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 850498578 0 0) 0x7f7260138b60 con 0x7f72600a98f0 2024-03-25T19:32:07.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 >> v1:172.21.15.19:6790/0 conn(0x7f72601430c0 legacy=0x7f7260133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:07.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 >> v1:172.21.15.188:6789/0 conn(0x7f726013f710 legacy=0x7f72601d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:07.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f725cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:32:07.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f725cff9640 1 monclient: found mon.a 2024-03-25T19:32:07.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f725cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:07.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f725cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:07.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7260134e70 con 0x7f72600a98f0 2024-03-25T19:32:07.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f725cff9640 10 monclient: _finish_auth 0 2024-03-25T19:32:07.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f725cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:32:07.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f725cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:07.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f7265a25640 5 monclient: authenticate success, global_id 4371 2024-03-25T19:32:07.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f7265a25640 10 monclient: _renew_subs 2024-03-25T19:32:07.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f7265a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:07.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f7265a25640 1 -- 172.21.15.19:0/4159816712 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72601350c0 con 0x7f72600a98f0 2024-03-25T19:32:07.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f7265a25640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:07.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f7265a25640 10 monclient: _renew_subs 2024-03-25T19:32:07.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f7265a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:07.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f7265a25640 1 -- 172.21.15.19:0/4159816712 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7260134800 con 0x7f72600a98f0 2024-03-25T19:32:07.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.188+0000 7f7265a25640 1 librados: init done 2024-03-25T19:32:07.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.189+0000 7f7265a25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:07.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.189+0000 7f7265a25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:07.192 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.189+0000 7f7265a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:07.192 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.189+0000 7f7265a25640 1 -- 172.21.15.19:0/4159816712 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f726014f330 con 0x7f72600a98f0 2024-03-25T19:32:07.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.520+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72500047e0 con 0x7f72600a98f0 2024-03-25T19:32:07.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.520+0000 7f725cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:07.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.520+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7250004a90 con 0x7f72600a98f0 2024-03-25T19:32:07.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.520+0000 7f725cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:07.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.520+0000 7f725cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:07.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.520+0000 7f725cff9640 10 monclient: dump: 2024-03-25T19:32:07.520 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:07.520 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:07.520 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:07.520 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:07.520 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:07.520 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:07.520 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:07.520 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:07.520 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:07.520 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:07.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.520+0000 7f725dffb640 4 set_mon_vals no callback set 2024-03-25T19:32:07.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.521+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f7250004cb0 con 0x7f72600a98f0 2024-03-25T19:32:07.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.522+0000 7f7257fff640 10 monclient: get_auth_request con 0x7f724006dd90 auth_method 0 2024-03-25T19:32:07.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.522+0000 7f725cff9640 10 client.4371.objecter ms_handle_connect 0x7f724006dd90 2024-03-25T19:32:07.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.527+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f72500546a0 con 0x7f72600a98f0 2024-03-25T19:32:07.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.527+0000 7f725cff9640 10 client.4371.objecter ms_dispatch 0x7f7260000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:07.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.527+0000 7f725cff9640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:07.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.527+0000 7f725cff9640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:07.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.527+0000 7f725cff9640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T19:32:07.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.527+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72500501b0 con 0x7f72600a98f0 2024-03-25T19:32:07.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.527+0000 7f725cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:07.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.527+0000 7f725cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:07.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.781+0000 7f7265a25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:07.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.781+0000 7f7265a25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:07.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.781+0000 7f7265a25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:07.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.781+0000 7f7265a25640 1 -- 172.21.15.19:0/4159816712 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72600abb30 con 0x7f72600a98f0 2024-03-25T19:32:07.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.812+0000 7f725cff9640 1 -- 172.21.15.19:0/4159816712 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f72500520e0 con 0x7f72600a98f0 2024-03-25T19:32:07.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.812+0000 7f725cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:07.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.812+0000 7f725cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:07.811 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:32:07.811 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:07.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 10 librados: watch_flush enter 2024-03-25T19:32:07.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 10 librados: watch_flush exit 2024-03-25T19:32:07.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T19:32:07.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 10 client.4371.objecter successfully canceled tick 2024-03-25T19:32:07.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 1 -- 172.21.15.19:0/4159816712 >> v1:172.21.15.188:6800/2207506229 conn(0x7f724006dd90 legacy=0x7f7240070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:07.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 10 monclient: shutdown 2024-03-25T19:32:07.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:07.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 1 -- 172.21.15.19:0/4159816712 >> v1:172.21.15.19:6789/0 conn(0x7f72600a98f0 legacy=0x7f7260137730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:07.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 1 -- 172.21.15.19:0/4159816712 shutdown_connections 2024-03-25T19:32:07.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 1 -- 172.21.15.19:0/4159816712 >> 172.21.15.19:0/4159816712 conn(0x7f72600b7ea0 msgr2=0x7f7260074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:07.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 1 -- 172.21.15.19:0/4159816712 shutdown_connections 2024-03-25T19:32:07.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 1 -- 172.21.15.19:0/4159816712 wait complete. 2024-03-25T19:32:07.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:07.814+0000 7f7265a25640 1 librados: shutdown 2024-03-25T19:32:07.826 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:12.190 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:34.919 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:12.203 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:34.932 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:12.224 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:34.953 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:12.241 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:03:34.944 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:12.257 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:03:34.960 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:12.259 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:34.989 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:12.274 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:35.004 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:12.292 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:03:34.995 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:12.313 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:35.042 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:14.632 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:14.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.745+0000 7fc389096640 10 monclient: get_monmap_and_config 2024-03-25T19:32:14.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.745+0000 7fc389096640 10 monclient: build_initial_monmap 2024-03-25T19:32:14.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.746+0000 7fc389096640 10 monclient: monmap: 2024-03-25T19:32:14.746 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:14.746 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:14.746 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:32:14.747245+0000 2024-03-25T19:32:14.746 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:32:14.747245+0000 2024-03-25T19:32:14.746 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:14.746 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:14.746 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:14.746 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:32:14.746 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:32:14.746 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.746+0000 7fc389096640 1 Processor -- start 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.747+0000 7fc389096640 1 -- start start 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.747+0000 7fc389096640 10 monclient: init 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.747+0000 7fc389096640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.747+0000 7fc389096640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.747+0000 7fc389096640 10 monclient(hunting): picked mon.noname-a con 0x7fc3840a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.747+0000 7fc389096640 10 monclient(hunting): picked mon.noname-b con 0x7fc38413f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.747+0000 7fc389096640 10 monclient(hunting): picked mon.noname-c con 0x7fc3841430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.747+0000 7fc389096640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc384142520 con 0x7fc3840a98f0 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.747+0000 7fc389096640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3840b28a0 con 0x7fc3841430c0 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.747+0000 7fc389096640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3840b2a30 con 0x7fc38413f710 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.747+0000 7fc389096640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.747+0000 7fc389096640 10 monclient(hunting): authenticate will time out at 6188.017697s 2024-03-25T19:32:14.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.748+0000 7fc38359e640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fc3841430c0 0x7fc38414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:47260/0 (socket says 172.21.15.19:47260) 2024-03-25T19:32:14.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.748+0000 7fc38359e640 1 -- 172.21.15.19:0/2780537091 learned_addr learned my addr 172.21.15.19:0/2780537091 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:14.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.815+0000 7fc38259c640 1 -- 172.21.15.19:0/2780537091 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162063359 0 0) 0x7fc384142520 con 0x7fc3840a98f0 2024-03-25T19:32:14.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.815+0000 7fc38259c640 10 monclient(hunting): _negotiate 2024-03-25T19:32:14.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.815+0000 7fc38259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:14.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.815+0000 7fc38259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:14.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.815+0000 7fc38259c640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:32:14.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.815+0000 7fc38259c640 1 -- 172.21.15.19:0/2780537091 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc370003520 con 0x7fc3840a98f0 2024-03-25T19:32:14.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 1 -- 172.21.15.19:0/2780537091 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3278557393 0 0) 0x7fc370003520 con 0x7fc3840a98f0 2024-03-25T19:32:14.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 1 -- 172.21.15.19:0/2780537091 >> v1:172.21.15.19:6790/0 conn(0x7fc38413f710 legacy=0x7fc3841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 1 -- 172.21.15.19:0/2780537091 >> v1:172.21.15.188:6789/0 conn(0x7fc3841430c0 legacy=0x7fc38414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 10 monclient: _finish_hunting 0 2024-03-25T19:32:14.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 1 monclient: found mon.noname-a 2024-03-25T19:32:14.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:14.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:32:14.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 1 -- 172.21.15.19:0/2780537091 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3840b44c0 con 0x7fc3840a98f0 2024-03-25T19:32:14.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 10 monclient: _finish_auth 0 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 10 monclient: _check_auth_tickets 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 1 -- 172.21.15.19:0/2780537091 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc374001690 con 0x7fc3840a98f0 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc38259c640 10 monclient: dump: 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:14.830 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:14.831 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:14.831 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:14.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc389096640 5 monclient: authenticate success, global_id 4374 2024-03-25T19:32:14.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.829+0000 7fc389096640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:14.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.844+0000 7fc38259c640 1 -- 172.21.15.19:0/2780537091 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3740018b0 con 0x7fc3840a98f0 2024-03-25T19:32:14.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.844+0000 7fc38259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:14.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.844+0000 7fc38259c640 1 -- 172.21.15.19:0/2780537091 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc374001c40 con 0x7fc3840a98f0 2024-03-25T19:32:14.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.844+0000 7fc38259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:14.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.844+0000 7fc38259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:14.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.844+0000 7fc38259c640 10 monclient: dump: 2024-03-25T19:32:14.844 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:14.844 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:14.844 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:14.844 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:14.845 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:14.845 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:14.845 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:14.845 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:14.845 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:14.845 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:14.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.844+0000 7fc389096640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:14.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.844+0000 7fc389096640 4 set_mon_vals no callback set 2024-03-25T19:32:14.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.844+0000 7fc389096640 10 monclient: shutdown 2024-03-25T19:32:14.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.844+0000 7fc389096640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:14.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.844+0000 7fc389096640 1 -- 172.21.15.19:0/2780537091 >> v1:172.21.15.19:6789/0 conn(0x7fc3840a98f0 legacy=0x7fc3840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.845+0000 7fc389096640 1 -- 172.21.15.19:0/2780537091 shutdown_connections 2024-03-25T19:32:14.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.845+0000 7fc389096640 1 -- 172.21.15.19:0/2780537091 >> 172.21.15.19:0/2780537091 conn(0x7fc3840b7ea0 msgr2=0x7fc3840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:14.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.845+0000 7fc389096640 1 -- 172.21.15.19:0/2780537091 shutdown_connections 2024-03-25T19:32:14.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.845+0000 7fc389096640 1 -- 172.21.15.19:0/2780537091 wait complete. 2024-03-25T19:32:14.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.845+0000 7fc389096640 10 monclient: build_initial_monmap 2024-03-25T19:32:14.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.845+0000 7fc389096640 10 monclient: monmap: 2024-03-25T19:32:14.847 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:14.847 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:14.847 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.846+0000 7fc389096640 1 librados: starting msgr at 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.846+0000 7fc389096640 1 librados: starting objecter 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.846+0000 7fc389096640 1 Processor -- start 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.846+0000 7fc389096640 1 -- start start 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.846+0000 7fc389096640 1 librados: setting wanted keys 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.846+0000 7fc389096640 1 librados: calling monclient init 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.846+0000 7fc389096640 10 monclient: init 2024-03-25T19:32:14.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.846+0000 7fc389096640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:14.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.846+0000 7fc389096640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:32:14.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.846+0000 7fc389096640 10 monclient(hunting): picked mon.noname-c con 0x7fc3840a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:14.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.847+0000 7fc389096640 10 monclient(hunting): picked mon.noname-b con 0x7fc38413f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:14.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.847+0000 7fc389096640 10 monclient(hunting): picked mon.noname-a con 0x7fc3841430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:14.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.847+0000 7fc389096640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc384135ac0 con 0x7fc3841430c0 2024-03-25T19:32:14.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.847+0000 7fc389096640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc384137030 con 0x7fc38413f710 2024-03-25T19:32:14.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.847+0000 7fc389096640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3841372a0 con 0x7fc3840a98f0 2024-03-25T19:32:14.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.847+0000 7fc389096640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:14.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.847+0000 7fc389096640 10 monclient(hunting): authenticate will time out at 6188.117033s 2024-03-25T19:32:14.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.861+0000 7fc38359e640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fc3841430c0 0x7fc3841e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:52142/0 (socket says 172.21.15.19:52142) 2024-03-25T19:32:14.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.861+0000 7fc38359e640 1 -- 172.21.15.19:0/1021027227 learned_addr learned my addr 172.21.15.19:0/1021027227 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:14.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.861+0000 7fc38359e640 1 -- 172.21.15.19:0/1021027227 >> v1:172.21.15.19:6789/0 conn(0x7fc3841430c0 legacy=0x7fc3841e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:32:14.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.861+0000 7fc38359e640 1 -- 172.21.15.19:0/1021027227 >> v1:172.21.15.19:6789/0 conn(0x7fc3841430c0 legacy=0x7fc3841e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:32:14.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.861+0000 7fc38359e640 1 --1- 172.21.15.19:0/1021027227 >> v1:172.21.15.19:6789/0 conn(0x7fc3841430c0 0x7fc3841e26b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T19:32:14.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.861+0000 7fc38359e640 1 --1- 172.21.15.19:0/1021027227 >> v1:172.21.15.19:6789/0 conn(0x7fc3841430c0 0x7fc3841e26b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T19:32:14.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.861+0000 7fc37b7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.19:6789/0 2024-03-25T19:32:14.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.875+0000 7fc37b7fe640 10 client.?.objecter ms_handle_connect 0x7fc38413f710 2024-03-25T19:32:14.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.875+0000 7fc37b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:14.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.882+0000 7fc37b7fe640 10 client.?.objecter ms_handle_connect 0x7fc3840a98f0 2024-03-25T19:32:14.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.882+0000 7fc37b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:14.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.887+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466865673 0 0) 0x7fc384137030 con 0x7fc38413f710 2024-03-25T19:32:14.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.887+0000 7fc37b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:32:14.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.887+0000 7fc37b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:14.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.887+0000 7fc37b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:14.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.887+0000 7fc37b7fe640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:32:14.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.887+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc364004eb0 con 0x7fc38413f710 2024-03-25T19:32:14.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.903+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2398686065 0 0) 0x7fc364004eb0 con 0x7fc38413f710 2024-03-25T19:32:14.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.903+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc384137030 con 0x7fc38413f710 2024-03-25T19:32:14.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.903+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc36c0018c0 con 0x7fc38413f710 2024-03-25T19:32:14.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.903+0000 7fc37b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.903+0000 7fc37b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.903+0000 7fc37b7fe640 10 monclient(hunting): dump: 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:14.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.903+0000 7fc37b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:14.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.919+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567810667 0 0) 0x7fc3841372a0 con 0x7fc3840a98f0 2024-03-25T19:32:14.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.919+0000 7fc37b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:32:14.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.919+0000 7fc37b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:14.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.919+0000 7fc37b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:14.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.919+0000 7fc37b7fe640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:32:14.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.919+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc364004eb0 con 0x7fc3840a98f0 2024-03-25T19:32:14.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3903498429 0 0) 0x7fc384137030 con 0x7fc38413f710 2024-03-25T19:32:14.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 >> v1:172.21.15.19:6790/0 conn(0x7fc3840a98f0 legacy=0x7fc384133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 >> v1:172.21.15.19:6789/0 conn(0x7fc3841430c0 legacy=0x7fc3841e26b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:32:14.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc37b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:32:14.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc37b7fe640 1 monclient: found mon.b 2024-03-25T19:32:14.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc37b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:14.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc37b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:14.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc384137580 con 0x7fc38413f710 2024-03-25T19:32:14.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc37b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:32:14.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc37b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:32:14.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc37b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:14.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc389096640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:32:14.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc389096640 10 monclient: _renew_subs 2024-03-25T19:32:14.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc389096640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:14.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.923+0000 7fc389096640 1 -- 172.21.15.19:0/1021027227 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3841363b0 con 0x7fc38413f710 2024-03-25T19:32:14.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.924+0000 7fc389096640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:14.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.924+0000 7fc389096640 10 monclient: _renew_subs 2024-03-25T19:32:14.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.924+0000 7fc389096640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:14.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.924+0000 7fc389096640 1 -- 172.21.15.19:0/1021027227 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3841353a0 con 0x7fc38413f710 2024-03-25T19:32:14.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.924+0000 7fc389096640 1 librados: init done 2024-03-25T19:32:14.926 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.924+0000 7fc389096640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:14.926 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.924+0000 7fc389096640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:14.926 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.924+0000 7fc389096640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:14.926 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:14.924+0000 7fc389096640 1 -- 172.21.15.19:0/1021027227 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc384130d90 con 0x7fc38413f710 2024-03-25T19:32:15.203 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.202+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc36c0035f0 con 0x7fc38413f710 2024-03-25T19:32:15.203 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.202+0000 7fc37b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:15.203 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.202+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc36c003820 con 0x7fc38413f710 2024-03-25T19:32:15.203 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.202+0000 7fc37b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:15.203 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.202+0000 7fc37b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:15.203 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.202+0000 7fc37b7fe640 10 monclient: dump: 2024-03-25T19:32:15.203 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:15.203 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:15.203 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:15.203 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:15.204 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:15.204 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:15.204 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:15.204 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:15.204 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:15.204 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:15.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.202+0000 7fc381d9b640 4 set_mon_vals no callback set 2024-03-25T19:32:15.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.203+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fc36c003a40 con 0x7fc38413f710 2024-03-25T19:32:15.205 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.204+0000 7fc382d9d640 10 monclient: get_auth_request con 0x7fc36406ddc0 auth_method 0 2024-03-25T19:32:15.205 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.204+0000 7fc37b7fe640 10 client.4366.objecter ms_handle_connect 0x7fc36406ddc0 2024-03-25T19:32:15.205 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.206+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fc36c053400 con 0x7fc38413f710 2024-03-25T19:32:15.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.206+0000 7fc37b7fe640 10 client.4366.objecter ms_dispatch 0x7fc384000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:15.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.206+0000 7fc37b7fe640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:15.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.206+0000 7fc37b7fe640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:15.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.206+0000 7fc37b7fe640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T19:32:15.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.208+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc36c04ef10 con 0x7fc38413f710 2024-03-25T19:32:15.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.208+0000 7fc37b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:15.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.208+0000 7fc37b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:15.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.469+0000 7fc389096640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:15.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.469+0000 7fc389096640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:15.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.469+0000 7fc389096640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:15.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.469+0000 7fc389096640 1 -- 172.21.15.19:0/1021027227 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc384135c70 con 0x7fc38413f710 2024-03-25T19:32:15.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.511+0000 7fc37b7fe640 1 -- 172.21.15.19:0/1021027227 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fc384135c70 con 0x7fc38413f710 2024-03-25T19:32:15.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.511+0000 7fc37b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:15.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.511+0000 7fc37b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:15.512 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:32:15.512 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:15.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 10 librados: watch_flush enter 2024-03-25T19:32:15.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 10 librados: watch_flush exit 2024-03-25T19:32:15.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T19:32:15.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 10 client.4366.objecter successfully canceled tick 2024-03-25T19:32:15.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 1 -- 172.21.15.19:0/1021027227 >> v1:172.21.15.188:6800/2207506229 conn(0x7fc36406ddc0 legacy=0x7fc364070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:15.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 10 monclient: shutdown 2024-03-25T19:32:15.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:15.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 1 -- 172.21.15.19:0/1021027227 >> v1:172.21.15.188:6789/0 conn(0x7fc38413f710 legacy=0x7fc3841379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:15.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 1 -- 172.21.15.19:0/1021027227 shutdown_connections 2024-03-25T19:32:15.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 1 -- 172.21.15.19:0/1021027227 >> 172.21.15.19:0/1021027227 conn(0x7fc3840b7ea0 msgr2=0x7fc384074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:15.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 1 -- 172.21.15.19:0/1021027227 shutdown_connections 2024-03-25T19:32:15.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 1 -- 172.21.15.19:0/1021027227 wait complete. 2024-03-25T19:32:15.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:15.514+0000 7fc389096640 1 librados: shutdown 2024-03-25T19:32:15.526 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:22.333 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:22.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.448+0000 7fda8c29a640 10 monclient: get_monmap_and_config 2024-03-25T19:32:22.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.448+0000 7fda8c29a640 10 monclient: build_initial_monmap 2024-03-25T19:32:22.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.448+0000 7fda8c29a640 10 monclient: monmap: 2024-03-25T19:32:22.448 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:22.448 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:22.448 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:32:22.449292+0000 2024-03-25T19:32:22.449 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:32:22.449292+0000 2024-03-25T19:32:22.449 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:22.449 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:22.449 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:22.449 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:32:22.449 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:32:22.449 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:22.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.448+0000 7fda8c29a640 1 Processor -- start 2024-03-25T19:32:22.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.448+0000 7fda8c29a640 1 -- start start 2024-03-25T19:32:22.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.448+0000 7fda8c29a640 10 monclient: init 2024-03-25T19:32:22.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.449+0000 7fda8c29a640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:22.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.449+0000 7fda8c29a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:22.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.449+0000 7fda8c29a640 10 monclient(hunting): picked mon.noname-c con 0x7fda840b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:22.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.449+0000 7fda8c29a640 10 monclient(hunting): picked mon.noname-b con 0x7fda840b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:22.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.449+0000 7fda8c29a640 10 monclient(hunting): picked mon.noname-a con 0x7fda8414be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:22.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.449+0000 7fda8c29a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda840af940 con 0x7fda8414be90 2024-03-25T19:32:22.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.449+0000 7fda8c29a640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda840b0a70 con 0x7fda840b4d60 2024-03-25T19:32:22.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.449+0000 7fda8c29a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda840b0d50 con 0x7fda840b1730 2024-03-25T19:32:22.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.449+0000 7fda8c29a640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:22.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.449+0000 7fda8c29a640 10 monclient(hunting): authenticate will time out at 6195.719893s 2024-03-25T19:32:22.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.453+0000 7fda8a837640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fda8414be90 0x7fda8414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:57590/0 (socket says 172.21.15.19:57590) 2024-03-25T19:32:22.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.453+0000 7fda8a837640 1 -- 172.21.15.19:0/3764479012 learned_addr learned my addr 172.21.15.19:0/3764479012 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:22.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.469+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73456319 0 0) 0x7fda840af940 con 0x7fda8414be90 2024-03-25T19:32:22.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.469+0000 7fda89034640 10 monclient(hunting): _negotiate 2024-03-25T19:32:22.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.469+0000 7fda89034640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:22.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.469+0000 7fda89034640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:22.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.470+0000 7fda89034640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:32:22.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.470+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda70003520 con 0x7fda8414be90 2024-03-25T19:32:22.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.473+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574873312 0 0) 0x7fda840b0d50 con 0x7fda840b1730 2024-03-25T19:32:22.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.473+0000 7fda89034640 10 monclient(hunting): _negotiate 2024-03-25T19:32:22.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.473+0000 7fda89034640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:22.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.473+0000 7fda89034640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:22.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.473+0000 7fda89034640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:32:22.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.473+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda840af940 con 0x7fda840b1730 2024-03-25T19:32:22.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.473+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098401894 0 0) 0x7fda840b0a70 con 0x7fda840b4d60 2024-03-25T19:32:22.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.473+0000 7fda89034640 10 monclient(hunting): _negotiate 2024-03-25T19:32:22.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.473+0000 7fda89034640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:22.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.473+0000 7fda89034640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:22.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.473+0000 7fda89034640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:32:22.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.473+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda840b0d50 con 0x7fda840b4d60 2024-03-25T19:32:22.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3431967784 0 0) 0x7fda70003520 con 0x7fda8414be90 2024-03-25T19:32:22.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 >> v1:172.21.15.19:6790/0 conn(0x7fda840b1730 legacy=0x7fda840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 >> v1:172.21.15.188:6789/0 conn(0x7fda840b4d60 legacy=0x7fda8413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 10 monclient: _finish_hunting 0 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 1 monclient: found mon.noname-a 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda840b1030 con 0x7fda8414be90 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 10 monclient: _finish_auth 0 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 10 monclient: _check_auth_tickets 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fda7c001a10 con 0x7fda8414be90 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda8c29a640 5 monclient: authenticate success, global_id 4377 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda8c29a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:22.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda89034640 10 monclient: dump: 2024-03-25T19:32:22.498 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:22.498 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:22.498 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:22.498 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:22.498 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:22.498 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:22.498 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:22.498 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:22.498 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:22.498 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:22.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.496+0000 7fda8c29a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:22.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.543+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda7c001b70 con 0x7fda8414be90 2024-03-25T19:32:22.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.543+0000 7fda89034640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:22.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.543+0000 7fda89034640 1 -- 172.21.15.19:0/3764479012 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fda7c001f80 con 0x7fda8414be90 2024-03-25T19:32:22.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.543+0000 7fda8c29a640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:22.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.543+0000 7fda8c29a640 4 set_mon_vals no callback set 2024-03-25T19:32:22.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.543+0000 7fda89034640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:22.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.543+0000 7fda89034640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:22.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.543+0000 7fda89034640 10 monclient: dump: 2024-03-25T19:32:22.544 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:22.544 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:22.544 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:22.544 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:22.544 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:22.544 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:22.544 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:22.544 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:22.544 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:22.544 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.544+0000 7fda8c29a640 10 monclient: shutdown 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.544+0000 7fda8c29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.544+0000 7fda8c29a640 1 -- 172.21.15.19:0/3764479012 >> v1:172.21.15.19:6789/0 conn(0x7fda8414be90 legacy=0x7fda8414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.544+0000 7fda8c29a640 1 -- 172.21.15.19:0/3764479012 shutdown_connections 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.544+0000 7fda8c29a640 1 -- 172.21.15.19:0/3764479012 >> 172.21.15.19:0/3764479012 conn(0x7fda840617a0 msgr2=0x7fda84061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.544+0000 7fda8c29a640 1 -- 172.21.15.19:0/3764479012 shutdown_connections 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.544+0000 7fda8c29a640 1 -- 172.21.15.19:0/3764479012 wait complete. 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.545+0000 7fda8c29a640 10 monclient: build_initial_monmap 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.545+0000 7fda8c29a640 10 monclient: monmap: 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:32:22.546 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.545+0000 7fda8c29a640 1 librados: starting msgr at 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.545+0000 7fda8c29a640 1 librados: starting objecter 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.546+0000 7fda8c29a640 1 Processor -- start 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.546+0000 7fda8c29a640 1 -- start start 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.546+0000 7fda8c29a640 1 librados: setting wanted keys 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.546+0000 7fda8c29a640 1 librados: calling monclient init 2024-03-25T19:32:22.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.546+0000 7fda8c29a640 10 monclient: init 2024-03-25T19:32:22.548 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.546+0000 7fda8c29a640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:22.548 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.546+0000 7fda8c29a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:22.548 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.547+0000 7fda8c29a640 10 monclient(hunting): picked mon.noname-a con 0x7fda840b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:22.548 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.547+0000 7fda8c29a640 10 monclient(hunting): picked mon.noname-b con 0x7fda840b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:22.548 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.547+0000 7fda8c29a640 10 monclient(hunting): picked mon.noname-c con 0x7fda8414be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:22.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.547+0000 7fda8c29a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda84137850 con 0x7fda840b1730 2024-03-25T19:32:22.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.547+0000 7fda8c29a640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda84137b30 con 0x7fda840b4d60 2024-03-25T19:32:22.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.547+0000 7fda8c29a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda84144720 con 0x7fda8414be90 2024-03-25T19:32:22.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.547+0000 7fda8c29a640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:22.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.547+0000 7fda8c29a640 10 monclient(hunting): authenticate will time out at 6195.817073s 2024-03-25T19:32:22.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.559+0000 7fda89835640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fda840b4d60 0x7fda84143900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:46342/0 (socket says 172.21.15.19:46342) 2024-03-25T19:32:22.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.559+0000 7fda89835640 1 -- 172.21.15.19:0/911865629 learned_addr learned my addr 172.21.15.19:0/911865629 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:22.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.562+0000 7fda7affd640 10 client.?.objecter ms_handle_connect 0x7fda840b4d60 2024-03-25T19:32:22.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.562+0000 7fda7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:22.566 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.566+0000 7fda7affd640 10 client.?.objecter ms_handle_connect 0x7fda8414be90 2024-03-25T19:32:22.566 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.566+0000 7fda7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:22.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.570+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267679910 0 0) 0x7fda84137b30 con 0x7fda840b4d60 2024-03-25T19:32:22.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.570+0000 7fda7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:22.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.570+0000 7fda7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:22.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.570+0000 7fda7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:22.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.570+0000 7fda7affd640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:32:22.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.570+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda68004eb0 con 0x7fda840b4d60 2024-03-25T19:32:22.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.579+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2386941227 0 0) 0x7fda68004eb0 con 0x7fda840b4d60 2024-03-25T19:32:22.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.579+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda84137b30 con 0x7fda840b4d60 2024-03-25T19:32:22.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.579+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fda74001a30 con 0x7fda840b4d60 2024-03-25T19:32:22.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.579+0000 7fda7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:22.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.579+0000 7fda7affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:22.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.579+0000 7fda7affd640 10 monclient(hunting): dump: 2024-03-25T19:32:22.579 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:22.580 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:22.580 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:22.580 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:22.580 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:22.580 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:22.580 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:22.580 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:22.580 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:22.580 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:22.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.579+0000 7fda7affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:22.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.579+0000 7fda7affd640 10 client.?.objecter ms_handle_connect 0x7fda840b1730 2024-03-25T19:32:22.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.579+0000 7fda7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:22.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.581+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809564794 0 0) 0x7fda84144720 con 0x7fda8414be90 2024-03-25T19:32:22.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.581+0000 7fda7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:22.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.581+0000 7fda7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:22.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.581+0000 7fda7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:22.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.581+0000 7fda7affd640 10 monclient(hunting): my global_id is 4352 2024-03-25T19:32:22.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.581+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda68004eb0 con 0x7fda8414be90 2024-03-25T19:32:22.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 98800702 0 0) 0x7fda84137b30 con 0x7fda840b4d60 2024-03-25T19:32:22.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 >> v1:172.21.15.19:6790/0 conn(0x7fda8414be90 legacy=0x7fda84144010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 >> v1:172.21.15.19:6789/0 conn(0x7fda840b1730 legacy=0x7fda84137f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda7affd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda7affd640 1 monclient: found mon.b 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda7affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda84136760 con 0x7fda840b4d60 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda7affd640 10 monclient: _finish_auth 0 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda7affd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda8c29a640 5 monclient: authenticate success, global_id 4372 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda8c29a640 10 monclient: _renew_subs 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda8c29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:22.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda8c29a640 1 -- 172.21.15.19:0/911865629 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda84144b90 con 0x7fda840b4d60 2024-03-25T19:32:22.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda8c29a640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:22.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda8c29a640 10 monclient: _renew_subs 2024-03-25T19:32:22.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda8c29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:22.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda8c29a640 1 -- 172.21.15.19:0/911865629 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda84138840 con 0x7fda840b4d60 2024-03-25T19:32:22.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.595+0000 7fda8c29a640 1 librados: init done 2024-03-25T19:32:22.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.596+0000 7fda8c29a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:22.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.596+0000 7fda8c29a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:22.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.596+0000 7fda8c29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:22.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.596+0000 7fda8c29a640 1 -- 172.21.15.19:0/911865629 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda840627b0 con 0x7fda840b4d60 2024-03-25T19:32:22.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.881+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda74001d10 con 0x7fda840b4d60 2024-03-25T19:32:22.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.881+0000 7fda7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:22.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.881+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fda74003a60 con 0x7fda840b4d60 2024-03-25T19:32:22.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.881+0000 7fda7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:22.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.881+0000 7fda7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:22.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.881+0000 7fda7affd640 10 monclient: dump: 2024-03-25T19:32:22.882 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:22.882 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:22.882 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:22.882 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:22.883 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:22.883 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:22.883 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:22.883 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:22.883 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:22.883 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:22.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.881+0000 7fda88833640 4 set_mon_vals no callback set 2024-03-25T19:32:22.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.882+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fda74003c80 con 0x7fda840b4d60 2024-03-25T19:32:22.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.883+0000 7fda8a036640 10 monclient: get_auth_request con 0x7fda6806dd90 auth_method 0 2024-03-25T19:32:22.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.883+0000 7fda7affd640 10 client.4372.objecter ms_handle_connect 0x7fda6806dd90 2024-03-25T19:32:22.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.887+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fda74054890 con 0x7fda840b4d60 2024-03-25T19:32:22.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.887+0000 7fda7affd640 10 client.4372.objecter ms_dispatch 0x7fda84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:22.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.887+0000 7fda7affd640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:22.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.887+0000 7fda7affd640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:22.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.887+0000 7fda7affd640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T19:32:22.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.887+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda740503a0 con 0x7fda840b4d60 2024-03-25T19:32:22.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.887+0000 7fda7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:22.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:22.887+0000 7fda7affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:23.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.121+0000 7fda8c29a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:23.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.121+0000 7fda8c29a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:23.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.121+0000 7fda8c29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:23.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.121+0000 7fda8c29a640 1 -- 172.21.15.19:0/911865629 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda8413c250 con 0x7fda840b4d60 2024-03-25T19:32:23.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.173+0000 7fda7affd640 1 -- 172.21.15.19:0/911865629 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fda740522d0 con 0x7fda840b4d60 2024-03-25T19:32:23.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.173+0000 7fda7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:23.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.173+0000 7fda7affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:23.173 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:32:23.173 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:23.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.175+0000 7fda8c29a640 10 librados: watch_flush enter 2024-03-25T19:32:23.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.175+0000 7fda8c29a640 10 librados: watch_flush exit 2024-03-25T19:32:23.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.175+0000 7fda8c29a640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T19:32:23.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.175+0000 7fda8c29a640 10 client.4372.objecter successfully canceled tick 2024-03-25T19:32:23.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.176+0000 7fda8c29a640 1 -- 172.21.15.19:0/911865629 >> v1:172.21.15.188:6800/2207506229 conn(0x7fda6806dd90 legacy=0x7fda68070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:23.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.176+0000 7fda8c29a640 10 monclient: shutdown 2024-03-25T19:32:23.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.176+0000 7fda8c29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:23.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.176+0000 7fda8c29a640 1 -- 172.21.15.19:0/911865629 >> v1:172.21.15.188:6789/0 conn(0x7fda840b4d60 legacy=0x7fda84143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:23.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.176+0000 7fda8c29a640 1 -- 172.21.15.19:0/911865629 shutdown_connections 2024-03-25T19:32:23.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.176+0000 7fda8c29a640 1 -- 172.21.15.19:0/911865629 >> 172.21.15.19:0/911865629 conn(0x7fda840617a0 msgr2=0x7fda840b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:23.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.176+0000 7fda8c29a640 1 -- 172.21.15.19:0/911865629 shutdown_connections 2024-03-25T19:32:23.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.176+0000 7fda8c29a640 1 -- 172.21.15.19:0/911865629 wait complete. 2024-03-25T19:32:23.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:23.176+0000 7fda8c29a640 1 librados: shutdown 2024-03-25T19:32:23.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:29.995 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:30.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.108+0000 7f55cb44e640 10 monclient: get_monmap_and_config 2024-03-25T19:32:30.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.108+0000 7f55cb44e640 10 monclient: build_initial_monmap 2024-03-25T19:32:30.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.108+0000 7f55cb44e640 10 monclient: monmap: 2024-03-25T19:32:30.108 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:30.108 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:30.108 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:32:30.109748+0000 2024-03-25T19:32:30.109 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:32:30.109748+0000 2024-03-25T19:32:30.109 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:30.109 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:30.109 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:30.109 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:32:30.109 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:32:30.109 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:30.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.109+0000 7f55cb44e640 1 Processor -- start 2024-03-25T19:32:30.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.109+0000 7f55cb44e640 1 -- start start 2024-03-25T19:32:30.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.109+0000 7f55cb44e640 10 monclient: init 2024-03-25T19:32:30.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.110+0000 7f55cb44e640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:30.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.110+0000 7f55cb44e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:30.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.110+0000 7f55cb44e640 10 monclient(hunting): picked mon.noname-c con 0x7f55c40a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:30.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.110+0000 7f55cb44e640 10 monclient(hunting): picked mon.noname-b con 0x7f55c413f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:30.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.110+0000 7f55cb44e640 10 monclient(hunting): picked mon.noname-a con 0x7f55c41430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:30.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.110+0000 7f55cb44e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c4142520 con 0x7f55c41430c0 2024-03-25T19:32:30.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.110+0000 7f55cb44e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c40b28a0 con 0x7f55c40a98f0 2024-03-25T19:32:30.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.110+0000 7f55cb44e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c40b2a30 con 0x7f55c413f710 2024-03-25T19:32:30.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.110+0000 7f55cb44e640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:30.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.110+0000 7f55cb44e640 10 monclient(hunting): authenticate will time out at 6203.380116s 2024-03-25T19:32:30.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.125+0000 7f55c89e9640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f55c413f710 0x7f55c41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:54226/0 (socket says 172.21.15.19:54226) 2024-03-25T19:32:30.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.125+0000 7f55c89e9640 1 -- 172.21.15.19:0/1034402395 learned_addr learned my addr 172.21.15.19:0/1034402395 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:30.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.138+0000 7f55bbfff640 1 -- 172.21.15.19:0/1034402395 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878303001 0 0) 0x7f55c40b2a30 con 0x7f55c413f710 2024-03-25T19:32:30.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.138+0000 7f55bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:30.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.138+0000 7f55bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:30.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.138+0000 7f55bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:30.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.138+0000 7f55bbfff640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:32:30.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.138+0000 7f55bbfff640 1 -- 172.21.15.19:0/1034402395 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b0003520 con 0x7f55c413f710 2024-03-25T19:32:30.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.143+0000 7f55bbfff640 1 -- 172.21.15.19:0/1034402395 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835116260 0 0) 0x7f55c40b28a0 con 0x7f55c40a98f0 2024-03-25T19:32:30.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.143+0000 7f55bbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:30.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.143+0000 7f55bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:30.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.143+0000 7f55bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:30.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.143+0000 7f55bbfff640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:32:30.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.143+0000 7f55bbfff640 1 -- 172.21.15.19:0/1034402395 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c40b2a30 con 0x7f55c40a98f0 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 1 -- 172.21.15.19:0/1034402395 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1914893746 0 0) 0x7f55b0003520 con 0x7f55c413f710 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 1 -- 172.21.15.19:0/1034402395 >> v1:172.21.15.188:6789/0 conn(0x7f55c40a98f0 legacy=0x7f55c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 1 -- 172.21.15.19:0/1034402395 >> v1:172.21.15.19:6789/0 conn(0x7f55c41430c0 legacy=0x7f55c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 1 monclient: found mon.noname-b 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 1 -- 172.21.15.19:0/1034402395 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c40b44c0 con 0x7f55c413f710 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 10 monclient: _finish_auth 0 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 1 -- 172.21.15.19:0/1034402395 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f55b4001690 con 0x7f55c413f710 2024-03-25T19:32:30.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55bbfff640 10 monclient: dump: 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55cb44e640 5 monclient: authenticate success, global_id 4355 2024-03-25T19:32:30.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.152+0000 7f55cb44e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:30.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.168+0000 7f55bbfff640 1 -- 172.21.15.19:0/1034402395 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55b40017f0 con 0x7f55c413f710 2024-03-25T19:32:30.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.168+0000 7f55bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:30.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.168+0000 7f55bbfff640 1 -- 172.21.15.19:0/1034402395 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f55b4001c00 con 0x7f55c413f710 2024-03-25T19:32:30.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.168+0000 7f55bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:30.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.168+0000 7f55bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:30.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.168+0000 7f55bbfff640 10 monclient: dump: 2024-03-25T19:32:30.169 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:30.169 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:30.169 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:30.169 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:30.169 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:30.169 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:30.169 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:30.169 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:30.169 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:30.169 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:30.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.168+0000 7f55cb44e640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:30.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.168+0000 7f55cb44e640 4 set_mon_vals no callback set 2024-03-25T19:32:30.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.169+0000 7f55cb44e640 10 monclient: shutdown 2024-03-25T19:32:30.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.169+0000 7f55cb44e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:30.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.169+0000 7f55cb44e640 1 -- 172.21.15.19:0/1034402395 >> v1:172.21.15.19:6790/0 conn(0x7f55c413f710 legacy=0x7f55c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:30.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.169+0000 7f55cb44e640 1 -- 172.21.15.19:0/1034402395 shutdown_connections 2024-03-25T19:32:30.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.169+0000 7f55cb44e640 1 -- 172.21.15.19:0/1034402395 >> 172.21.15.19:0/1034402395 conn(0x7f55c40b7ea0 msgr2=0x7f55c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:30.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.169+0000 7f55cb44e640 1 -- 172.21.15.19:0/1034402395 shutdown_connections 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.169+0000 7f55cb44e640 1 -- 172.21.15.19:0/1034402395 wait complete. 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.170+0000 7f55cb44e640 10 monclient: build_initial_monmap 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.170+0000 7f55cb44e640 10 monclient: monmap: 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:32:30.171 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.171+0000 7f55cb44e640 1 librados: starting msgr at 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.171+0000 7f55cb44e640 1 librados: starting objecter 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.171+0000 7f55cb44e640 1 Processor -- start 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.171+0000 7f55cb44e640 1 -- start start 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.171+0000 7f55cb44e640 1 librados: setting wanted keys 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.171+0000 7f55cb44e640 1 librados: calling monclient init 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.171+0000 7f55cb44e640 10 monclient: init 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.172+0000 7f55cb44e640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.172+0000 7f55cb44e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.172+0000 7f55cb44e640 10 monclient(hunting): picked mon.noname-a con 0x7f55c40a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.172+0000 7f55cb44e640 10 monclient(hunting): picked mon.noname-b con 0x7f55c413f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.172+0000 7f55cb44e640 10 monclient(hunting): picked mon.noname-c con 0x7f55c41430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.172+0000 7f55cb44e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c41370a0 con 0x7f55c40a98f0 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.172+0000 7f55cb44e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c4135090 con 0x7f55c413f710 2024-03-25T19:32:30.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.172+0000 7f55cb44e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c4135220 con 0x7f55c41430c0 2024-03-25T19:32:30.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.172+0000 7f55cb44e640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:30.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.172+0000 7f55cb44e640 10 monclient(hunting): authenticate will time out at 6203.442352s 2024-03-25T19:32:30.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.176+0000 7f55c89e9640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f55c413f710 0x7f55c41d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:46506/0 (socket says 172.21.15.19:46506) 2024-03-25T19:32:30.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.176+0000 7f55c89e9640 1 -- 172.21.15.19:0/1487873870 learned_addr learned my addr 172.21.15.19:0/1487873870 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:30.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.177+0000 7f55b9ffb640 10 client.?.objecter ms_handle_connect 0x7f55c413f710 2024-03-25T19:32:30.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.177+0000 7f55b9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:30.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.183+0000 7f55b9ffb640 10 client.?.objecter ms_handle_connect 0x7f55c41430c0 2024-03-25T19:32:30.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.183+0000 7f55b9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:30.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.191+0000 7f55b9ffb640 10 client.?.objecter ms_handle_connect 0x7f55c40a98f0 2024-03-25T19:32:30.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.191+0000 7f55b9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:30.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.196+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657924297 0 0) 0x7f55c4135090 con 0x7f55c413f710 2024-03-25T19:32:30.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.196+0000 7f55b9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:30.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.196+0000 7f55b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:30.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.196+0000 7f55b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:30.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.196+0000 7f55b9ffb640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:32:30.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.196+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a8004eb0 con 0x7f55c413f710 2024-03-25T19:32:30.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.201+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124217253 0 0) 0x7f55c4135220 con 0x7f55c41430c0 2024-03-25T19:32:30.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.201+0000 7f55b9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:30.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.201+0000 7f55b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:30.202 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.201+0000 7f55b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:30.202 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.201+0000 7f55b9ffb640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:32:30.202 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.201+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c4135090 con 0x7f55c41430c0 2024-03-25T19:32:30.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.206+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123903079 0 0) 0x7f55c41370a0 con 0x7f55c40a98f0 2024-03-25T19:32:30.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.206+0000 7f55b9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:30.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.206+0000 7f55b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:30.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.206+0000 7f55b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:30.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.206+0000 7f55b9ffb640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:32:30.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.206+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c4135220 con 0x7f55c40a98f0 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.213+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 453101546 0 0) 0x7f55c4135090 con 0x7f55c41430c0 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.213+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55c41370a0 con 0x7f55c41430c0 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.213+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f55bc001b10 con 0x7f55c41430c0 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.213+0000 7f55b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.213+0000 7f55b9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.213+0000 7f55b9ffb640 10 monclient(hunting): dump: 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:30.213 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:30.214 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:30.214 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:30.214 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:30.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.213+0000 7f55b9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:30.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.220+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1513123285 0 0) 0x7f55c4135220 con 0x7f55c40a98f0 2024-03-25T19:32:30.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.220+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55c4135090 con 0x7f55c40a98f0 2024-03-25T19:32:30.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.220+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f55c0001930 con 0x7f55c40a98f0 2024-03-25T19:32:30.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.220+0000 7f55b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:30.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.220+0000 7f55b9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:30.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.220+0000 7f55b9ffb640 10 monclient(hunting): dump: 2024-03-25T19:32:30.220 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:30.220 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:30.220 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.221+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 374878756 0 0) 0x7f55a8004eb0 con 0x7f55c413f710 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.221+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55c4135220 con 0x7f55c413f710 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.221+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f55b4001bc0 con 0x7f55c413f710 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.221+0000 7f55b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.221+0000 7f55b9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:30.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.221+0000 7f55b9ffb640 10 monclient(hunting): dump: 2024-03-25T19:32:30.222 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:30.222 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:30.222 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:30.222 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:30.222 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:30.222 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:30.222 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:30.222 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:30.222 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:30.222 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1601421863 0 0) 0x7f55c4135090 con 0x7f55c40a98f0 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 >> v1:172.21.15.19:6790/0 conn(0x7f55c41430c0 legacy=0x7f55c41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 >> v1:172.21.15.188:6789/0 conn(0x7f55c413f710 legacy=0x7f55c41d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55b9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55b9ffb640 1 monclient: found mon.a 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55b9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c41d5000 con 0x7f55c40a98f0 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55b9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55b9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55cb44e640 5 monclient: authenticate success, global_id 4386 2024-03-25T19:32:30.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55cb44e640 10 monclient: _renew_subs 2024-03-25T19:32:30.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55cb44e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:30.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55cb44e640 1 -- 172.21.15.19:0/1487873870 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c41d52b0 con 0x7f55c40a98f0 2024-03-25T19:32:30.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55cb44e640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:30.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55cb44e640 10 monclient: _renew_subs 2024-03-25T19:32:30.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55cb44e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:30.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55cb44e640 1 -- 172.21.15.19:0/1487873870 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c4135910 con 0x7f55c40a98f0 2024-03-25T19:32:30.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.232+0000 7f55cb44e640 1 librados: init done 2024-03-25T19:32:30.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.234+0000 7f55cb44e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:30.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.234+0000 7f55cb44e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:30.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.234+0000 7f55cb44e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:30.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.234+0000 7f55cb44e640 1 -- 172.21.15.19:0/1487873870 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c40627b0 con 0x7f55c40a98f0 2024-03-25T19:32:30.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.552+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55c0003610 con 0x7f55c40a98f0 2024-03-25T19:32:30.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.552+0000 7f55b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:30.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.552+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f55c0001d80 con 0x7f55c40a98f0 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.552+0000 7f55b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.552+0000 7f55b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.552+0000 7f55b9ffb640 10 monclient: dump: 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.552+0000 7f55bb7fe640 4 set_mon_vals no callback set 2024-03-25T19:32:30.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.553+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f55c0019850 con 0x7f55c40a98f0 2024-03-25T19:32:30.554 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.553+0000 7f55c89e9640 10 monclient: get_auth_request con 0x7f55a806e160 auth_method 0 2024-03-25T19:32:30.554 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.554+0000 7f55b9ffb640 10 client.4386.objecter ms_handle_connect 0x7f55a806e160 2024-03-25T19:32:30.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.562+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f55c0053500 con 0x7f55c40a98f0 2024-03-25T19:32:30.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.562+0000 7f55b9ffb640 10 client.4386.objecter ms_dispatch 0x7f55c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:30.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.562+0000 7f55b9ffb640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:30.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.562+0000 7f55b9ffb640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:30.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.562+0000 7f55b9ffb640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-25T19:32:30.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.562+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55c004ee40 con 0x7f55c40a98f0 2024-03-25T19:32:30.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.562+0000 7f55b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:30.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.562+0000 7f55b9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:30.821 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.821+0000 7f55cb44e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:30.821 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.821+0000 7f55cb44e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:30.821 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.821+0000 7f55cb44e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:30.821 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.821+0000 7f55cb44e640 1 -- 172.21.15.19:0/1487873870 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55c413bfe0 con 0x7f55c40a98f0 2024-03-25T19:32:30.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.854+0000 7f55b9ffb640 1 -- 172.21.15.19:0/1487873870 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f55c0050f40 con 0x7f55c40a98f0 2024-03-25T19:32:30.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.854+0000 7f55b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:30.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.854+0000 7f55b9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:30.855 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:32:30.855 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:30.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 10 librados: watch_flush enter 2024-03-25T19:32:30.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 10 librados: watch_flush exit 2024-03-25T19:32:30.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-25T19:32:30.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 10 client.4386.objecter successfully canceled tick 2024-03-25T19:32:30.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 1 -- 172.21.15.19:0/1487873870 >> v1:172.21.15.188:6800/2207506229 conn(0x7f55a806e160 legacy=0x7f55a8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:30.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 10 monclient: shutdown 2024-03-25T19:32:30.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:30.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 1 -- 172.21.15.19:0/1487873870 >> v1:172.21.15.19:6789/0 conn(0x7f55c40a98f0 legacy=0x7f55c41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:30.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 1 -- 172.21.15.19:0/1487873870 shutdown_connections 2024-03-25T19:32:30.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 1 -- 172.21.15.19:0/1487873870 >> 172.21.15.19:0/1487873870 conn(0x7f55c40b7ea0 msgr2=0x7f55c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:30.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 1 -- 172.21.15.19:0/1487873870 shutdown_connections 2024-03-25T19:32:30.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 1 -- 172.21.15.19:0/1487873870 wait complete. 2024-03-25T19:32:30.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:30.857+0000 7f55cb44e640 1 librados: shutdown 2024-03-25T19:32:30.869 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:32.211 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:54.940 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:32.262 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:54.992 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:32.294 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:55.023 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:32.316 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:03:55.019 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:32.322 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:03:55.025 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:32.331 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:55.061 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:32.342 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:55.071 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:32.345 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:03:55.048 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:32.370 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:55.100 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:37.675 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:37.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.798+0000 7f582748d640 10 monclient: get_monmap_and_config 2024-03-25T19:32:37.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.798+0000 7f582748d640 10 monclient: build_initial_monmap 2024-03-25T19:32:37.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.798+0000 7f582748d640 10 monclient: monmap: 2024-03-25T19:32:37.798 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:37.798 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:37.799 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:32:37.799618+0000 2024-03-25T19:32:37.799 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:32:37.799618+0000 2024-03-25T19:32:37.799 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:37.799 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:37.799 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:37.799 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:32:37.799 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:32:37.799 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:37.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.798+0000 7f582748d640 1 Processor -- start 2024-03-25T19:32:37.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.799+0000 7f582748d640 1 -- start start 2024-03-25T19:32:37.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.799+0000 7f582748d640 10 monclient: init 2024-03-25T19:32:37.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.799+0000 7f582748d640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:37.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.799+0000 7f582748d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:32:37.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.799+0000 7f582748d640 10 monclient(hunting): picked mon.noname-b con 0x7f58200b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:37.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.799+0000 7f582748d640 10 monclient(hunting): picked mon.noname-c con 0x7f58200b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:37.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.799+0000 7f582748d640 10 monclient(hunting): picked mon.noname-a con 0x7f582014be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:37.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.799+0000 7f582748d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58200af940 con 0x7f582014be90 2024-03-25T19:32:37.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.799+0000 7f582748d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58200b0a70 con 0x7f58200b1730 2024-03-25T19:32:37.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.799+0000 7f582748d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58200b0d50 con 0x7f58200b4d60 2024-03-25T19:32:37.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.799+0000 7f582748d640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:37.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.799+0000 7f582748d640 10 monclient(hunting): authenticate will time out at 6211.069801s 2024-03-25T19:32:37.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.807+0000 7f5824a28640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f58200b1730 0x7f58200b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:51128/0 (socket says 172.21.15.19:51128) 2024-03-25T19:32:37.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.807+0000 7f5824a28640 1 -- 172.21.15.19:0/56544777 learned_addr learned my addr 172.21.15.19:0/56544777 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:37.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.820+0000 7f5817fff640 1 -- 172.21.15.19:0/56544777 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890688126 0 0) 0x7f58200b0a70 con 0x7f58200b1730 2024-03-25T19:32:37.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.820+0000 7f5817fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:37.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.820+0000 7f5817fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:37.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.820+0000 7f5817fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:37.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.820+0000 7f5817fff640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:32:37.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.820+0000 7f5817fff640 1 -- 172.21.15.19:0/56544777 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f580c003520 con 0x7f58200b1730 2024-03-25T19:32:37.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 1 -- 172.21.15.19:0/56544777 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1156791694 0 0) 0x7f580c003520 con 0x7f58200b1730 2024-03-25T19:32:37.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 1 -- 172.21.15.19:0/56544777 >> v1:172.21.15.19:6790/0 conn(0x7f58200b4d60 legacy=0x7f582013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 1 -- 172.21.15.19:0/56544777 >> v1:172.21.15.19:6789/0 conn(0x7f582014be90 legacy=0x7f582014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 1 monclient: found mon.noname-c 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 1 -- 172.21.15.19:0/56544777 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58200b1030 con 0x7f58200b1730 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 10 monclient: _finish_auth 0 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 1 -- 172.21.15.19:0/56544777 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5810001690 con 0x7f58200b1730 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f5817fff640 10 monclient: dump: 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:37.842 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:37.843 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:37.843 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:37.843 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:37.843 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:37.843 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:37.843 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:37.843 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:37.843 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:37.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f582748d640 5 monclient: authenticate success, global_id 4381 2024-03-25T19:32:37.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.841+0000 7f582748d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:37.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f5817fff640 1 -- 172.21.15.19:0/56544777 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58100017f0 con 0x7f58200b1730 2024-03-25T19:32:37.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f5817fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:37.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f5817fff640 1 -- 172.21.15.19:0/56544777 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5810001c00 con 0x7f58200b1730 2024-03-25T19:32:37.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f582748d640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:37.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f582748d640 4 set_mon_vals no callback set 2024-03-25T19:32:37.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f5817fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:37.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f5817fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:37.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f5817fff640 10 monclient: dump: 2024-03-25T19:32:37.858 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:37.858 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:37.858 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:37.858 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:37.859 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:37.859 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:37.859 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:37.859 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:37.859 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:37.859 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:37.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f582748d640 10 monclient: shutdown 2024-03-25T19:32:37.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f582748d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:37.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f582748d640 1 -- 172.21.15.19:0/56544777 >> v1:172.21.15.188:6789/0 conn(0x7f58200b1730 legacy=0x7f58200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f582748d640 1 -- 172.21.15.19:0/56544777 shutdown_connections 2024-03-25T19:32:37.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f582748d640 1 -- 172.21.15.19:0/56544777 >> 172.21.15.19:0/56544777 conn(0x7f58200617a0 msgr2=0x7f5820061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.858+0000 7f582748d640 1 -- 172.21.15.19:0/56544777 shutdown_connections 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.859+0000 7f582748d640 1 -- 172.21.15.19:0/56544777 wait complete. 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.859+0000 7f582748d640 10 monclient: build_initial_monmap 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.859+0000 7f582748d640 10 monclient: monmap: 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:37.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.860+0000 7f582748d640 1 librados: starting msgr at 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.860+0000 7f582748d640 1 librados: starting objecter 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.860+0000 7f582748d640 1 Processor -- start 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.860+0000 7f582748d640 1 -- start start 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.860+0000 7f582748d640 1 librados: setting wanted keys 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.860+0000 7f582748d640 1 librados: calling monclient init 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.860+0000 7f582748d640 10 monclient: init 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.860+0000 7f582748d640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.860+0000 7f582748d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.860+0000 7f582748d640 10 monclient(hunting): picked mon.noname-c con 0x7f58200b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.861+0000 7f582748d640 10 monclient(hunting): picked mon.noname-a con 0x7f58200b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.861+0000 7f582748d640 10 monclient(hunting): picked mon.noname-b con 0x7f582014be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.861+0000 7f582748d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5820135810 con 0x7f58200b4d60 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.861+0000 7f582748d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5820144c10 con 0x7f582014be90 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.861+0000 7f582748d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5820144ef0 con 0x7f58200b1730 2024-03-25T19:32:37.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.861+0000 7f582748d640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:37.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.861+0000 7f582748d640 10 monclient(hunting): authenticate will time out at 6211.131098s 2024-03-25T19:32:37.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.874+0000 7f5825a2a640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f582014be90 0x7f5820135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:51158/0 (socket says 172.21.15.19:51158) 2024-03-25T19:32:37.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.874+0000 7f5825a2a640 1 -- 172.21.15.19:0/4053328320 learned_addr learned my addr 172.21.15.19:0/4053328320 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:37.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.875+0000 7f5815ffb640 10 client.?.objecter ms_handle_connect 0x7f582014be90 2024-03-25T19:32:37.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.875+0000 7f5815ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:37.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.880+0000 7f5815ffb640 10 client.?.objecter ms_handle_connect 0x7f58200b4d60 2024-03-25T19:32:37.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.880+0000 7f5815ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:37.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.883+0000 7f5815ffb640 10 client.?.objecter ms_handle_connect 0x7f58200b1730 2024-03-25T19:32:37.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.883+0000 7f5815ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:37.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.888+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143317414 0 0) 0x7f5820135810 con 0x7f58200b4d60 2024-03-25T19:32:37.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.888+0000 7f5815ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:37.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.888+0000 7f5815ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:37.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.888+0000 7f5815ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:37.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.888+0000 7f5815ffb640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:32:37.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.888+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5804004eb0 con 0x7f58200b4d60 2024-03-25T19:32:37.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.898+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220545497 0 0) 0x7f5820144ef0 con 0x7f58200b1730 2024-03-25T19:32:37.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.898+0000 7f5815ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:37.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.898+0000 7f5815ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:37.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.898+0000 7f5815ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:37.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.898+0000 7f5815ffb640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:32:37.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.898+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5820135810 con 0x7f58200b1730 2024-03-25T19:32:37.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.898+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179947156 0 0) 0x7f5820144c10 con 0x7f582014be90 2024-03-25T19:32:37.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.898+0000 7f5815ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:37.899 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.898+0000 7f5815ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:37.899 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.898+0000 7f5815ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:37.899 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.898+0000 7f5815ffb640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:32:37.899 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.898+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5820144ef0 con 0x7f582014be90 2024-03-25T19:32:37.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.901+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 280574740 0 0) 0x7f5804004eb0 con 0x7f58200b4d60 2024-03-25T19:32:37.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.901+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5820144c10 con 0x7f58200b4d60 2024-03-25T19:32:37.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.901+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5810001b00 con 0x7f58200b4d60 2024-03-25T19:32:37.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.901+0000 7f5815ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:37.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.901+0000 7f5815ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:37.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.901+0000 7f5815ffb640 10 monclient(hunting): dump: 2024-03-25T19:32:37.901 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:37.901 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:37.901 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:37.901 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:37.901 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:37.901 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:37.902 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:37.902 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:37.902 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:37.902 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:37.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.901+0000 7f5815ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:37.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2192924714 0 0) 0x7f5820144c10 con 0x7f58200b4d60 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 >> v1:172.21.15.19:6790/0 conn(0x7f58200b1730 legacy=0x7f58201d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 >> v1:172.21.15.188:6789/0 conn(0x7f582014be90 legacy=0x7f5820135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f5815ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f5815ffb640 1 monclient: found mon.a 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f5815ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f5815ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5820140c10 con 0x7f58200b4d60 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f5815ffb640 10 monclient: _finish_auth 0 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f5815ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f5815ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f582748d640 5 monclient: authenticate success, global_id 4392 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f582748d640 10 monclient: _renew_subs 2024-03-25T19:32:37.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f582748d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:37.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f582748d640 1 -- 172.21.15.19:0/4053328320 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5820137d50 con 0x7f58200b4d60 2024-03-25T19:32:37.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f582748d640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:37.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f582748d640 10 monclient: _renew_subs 2024-03-25T19:32:37.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f582748d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:37.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f582748d640 1 -- 172.21.15.19:0/4053328320 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58201407d0 con 0x7f58200b4d60 2024-03-25T19:32:37.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.921+0000 7f582748d640 1 librados: init done 2024-03-25T19:32:37.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.922+0000 7f582748d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:37.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.922+0000 7f582748d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:37.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.922+0000 7f582748d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:37.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:37.922+0000 7f582748d640 1 -- 172.21.15.19:0/4053328320 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5820141e30 con 0x7f58200b4d60 2024-03-25T19:32:38.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.189+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5810004a90 con 0x7f58200b4d60 2024-03-25T19:32:38.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.189+0000 7f5815ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:38.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.189+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5810004da0 con 0x7f58200b4d60 2024-03-25T19:32:38.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.189+0000 7f5815ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.189+0000 7f5815ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.189+0000 7f5815ffb640 10 monclient: dump: 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:38.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.189+0000 7f58177fe640 4 set_mon_vals no callback set 2024-03-25T19:32:38.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.190+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f581001ad70 con 0x7f58200b4d60 2024-03-25T19:32:38.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.191+0000 7f5825229640 10 monclient: get_auth_request con 0x7f580406ddf0 auth_method 0 2024-03-25T19:32:38.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.191+0000 7f5815ffb640 10 client.4392.objecter ms_handle_connect 0x7f580406ddf0 2024-03-25T19:32:38.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.195+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f5810054820 con 0x7f58200b4d60 2024-03-25T19:32:38.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.195+0000 7f5815ffb640 10 client.4392.objecter ms_dispatch 0x7f5820000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:38.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.195+0000 7f5815ffb640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:38.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.195+0000 7f5815ffb640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:38.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.195+0000 7f5815ffb640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-25T19:32:38.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.195+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5810050330 con 0x7f58200b4d60 2024-03-25T19:32:38.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.195+0000 7f5815ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:38.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.195+0000 7f5815ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:38.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.423+0000 7f582748d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:38.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.423+0000 7f582748d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:38.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.423+0000 7f582748d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:32:38.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.423+0000 7f582748d640 1 -- 172.21.15.19:0/4053328320 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58201380a0 con 0x7f58200b4d60 2024-03-25T19:32:38.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.461+0000 7f5815ffb640 1 -- 172.21.15.19:0/4053328320 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f58201380a0 con 0x7f58200b4d60 2024-03-25T19:32:38.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.461+0000 7f5815ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:38.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.461+0000 7f5815ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:38.461 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:32:38.461 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:38.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 10 librados: watch_flush enter 2024-03-25T19:32:38.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 10 librados: watch_flush exit 2024-03-25T19:32:38.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-25T19:32:38.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 10 client.4392.objecter successfully canceled tick 2024-03-25T19:32:38.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 1 -- 172.21.15.19:0/4053328320 >> v1:172.21.15.188:6800/2207506229 conn(0x7f580406ddf0 legacy=0x7f58040702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:38.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 10 monclient: shutdown 2024-03-25T19:32:38.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:38.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 1 -- 172.21.15.19:0/4053328320 >> v1:172.21.15.19:6789/0 conn(0x7f58200b4d60 legacy=0x7f58201349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:38.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 1 -- 172.21.15.19:0/4053328320 shutdown_connections 2024-03-25T19:32:38.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 1 -- 172.21.15.19:0/4053328320 >> 172.21.15.19:0/4053328320 conn(0x7f58200617a0 msgr2=0x7f58200b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:38.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 1 -- 172.21.15.19:0/4053328320 shutdown_connections 2024-03-25T19:32:38.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 1 -- 172.21.15.19:0/4053328320 wait complete. 2024-03-25T19:32:38.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:38.463+0000 7f582748d640 1 librados: shutdown 2024-03-25T19:32:38.475 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:45.281 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:45.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.401+0000 7f75d0cd9640 10 monclient: get_monmap_and_config 2024-03-25T19:32:45.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.401+0000 7f75d0cd9640 10 monclient: build_initial_monmap 2024-03-25T19:32:45.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.402+0000 7f75d0cd9640 10 monclient: monmap: 2024-03-25T19:32:45.402 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:45.402 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:45.402 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:32:45.403201+0000 2024-03-25T19:32:45.402 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:32:45.403201+0000 2024-03-25T19:32:45.402 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:45.402 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:45.402 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:45.402 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:32:45.402 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:32:45.403 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:45.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.402+0000 7f75d0cd9640 1 Processor -- start 2024-03-25T19:32:45.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.402+0000 7f75d0cd9640 1 -- start start 2024-03-25T19:32:45.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.402+0000 7f75d0cd9640 10 monclient: init 2024-03-25T19:32:45.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.403+0000 7f75d0cd9640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:45.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.403+0000 7f75d0cd9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:32:45.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.403+0000 7f75d0cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f75cc06ab80 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:45.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.403+0000 7f75d0cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f75cc06af60 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:45.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.403+0000 7f75d0cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f75cc0b00b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:45.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.403+0000 7f75d0cd9640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0b0500 con 0x7f75cc06af60 2024-03-25T19:32:45.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.403+0000 7f75d0cd9640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc13ffa0 con 0x7f75cc06ab80 2024-03-25T19:32:45.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.403+0000 7f75d0cd9640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0af520 con 0x7f75cc0b00b0 2024-03-25T19:32:45.405 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.403+0000 7f75d0cd9640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:45.405 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.403+0000 7f75d0cd9640 10 monclient(hunting): authenticate will time out at 6218.673565s 2024-03-25T19:32:45.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.413+0000 7f75cad9d640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f75cc0b00b0 0x7f75cc14dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:51666/0 (socket says 172.21.15.19:51666) 2024-03-25T19:32:45.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.413+0000 7f75cad9d640 1 -- 172.21.15.19:0/935470727 learned_addr learned my addr 172.21.15.19:0/935470727 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:45.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.426+0000 7f75c9d9b640 1 -- 172.21.15.19:0/935470727 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689680657 0 0) 0x7f75cc0af520 con 0x7f75cc0b00b0 2024-03-25T19:32:45.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.427+0000 7f75c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:32:45.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.427+0000 7f75c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:45.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.427+0000 7f75c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:45.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.427+0000 7f75c9d9b640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:32:45.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.427+0000 7f75c9d9b640 1 -- 172.21.15.19:0/935470727 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8003520 con 0x7f75cc0b00b0 2024-03-25T19:32:45.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.431+0000 7f75c9d9b640 1 -- 172.21.15.19:0/935470727 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402231480 0 0) 0x7f75cc13ffa0 con 0x7f75cc06ab80 2024-03-25T19:32:45.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.431+0000 7f75c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:32:45.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.431+0000 7f75c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:45.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.431+0000 7f75c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:45.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.431+0000 7f75c9d9b640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:32:45.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.431+0000 7f75c9d9b640 1 -- 172.21.15.19:0/935470727 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75cc0af520 con 0x7f75cc06ab80 2024-03-25T19:32:45.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 1 -- 172.21.15.19:0/935470727 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3389458207 0 0) 0x7f75b8003520 con 0x7f75cc0b00b0 2024-03-25T19:32:45.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 1 -- 172.21.15.19:0/935470727 >> v1:172.21.15.188:6789/0 conn(0x7f75cc06ab80 legacy=0x7f75cc0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:45.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 1 -- 172.21.15.19:0/935470727 >> v1:172.21.15.19:6789/0 conn(0x7f75cc06af60 legacy=0x7f75cc13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:45.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:32:45.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 1 monclient: found mon.noname-b 2024-03-25T19:32:45.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:45.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:32:45.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 1 -- 172.21.15.19:0/935470727 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc072890 con 0x7f75cc0b00b0 2024-03-25T19:32:45.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 1 -- 172.21.15.19:0/935470727 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f75c4001a10 con 0x7f75cc0b00b0 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75c9d9b640 10 monclient: dump: 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:45.444 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:45.445 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:45.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75d0cd9640 5 monclient: authenticate success, global_id 4367 2024-03-25T19:32:45.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.443+0000 7f75d0cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:45.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75c9d9b640 1 -- 172.21.15.19:0/935470727 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75c4001b70 con 0x7f75cc0b00b0 2024-03-25T19:32:45.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:45.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75c9d9b640 1 -- 172.21.15.19:0/935470727 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f75c4001f80 con 0x7f75cc0b00b0 2024-03-25T19:32:45.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75d0cd9640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:45.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75d0cd9640 4 set_mon_vals no callback set 2024-03-25T19:32:45.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:45.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75c9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:45.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75c9d9b640 10 monclient: dump: 2024-03-25T19:32:45.455 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:45.455 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:45.455 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:45.455 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:45.456 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:45.456 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:45.456 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:45.456 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:45.456 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:45.456 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:45.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75d0cd9640 10 monclient: shutdown 2024-03-25T19:32:45.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75d0cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:45.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75d0cd9640 1 -- 172.21.15.19:0/935470727 >> v1:172.21.15.19:6790/0 conn(0x7f75cc0b00b0 legacy=0x7f75cc14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:45.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75d0cd9640 1 -- 172.21.15.19:0/935470727 shutdown_connections 2024-03-25T19:32:45.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75d0cd9640 1 -- 172.21.15.19:0/935470727 >> 172.21.15.19:0/935470727 conn(0x7f75cc0b4a80 msgr2=0x7f75cc0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:45.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.455+0000 7f75d0cd9640 1 -- 172.21.15.19:0/935470727 shutdown_connections 2024-03-25T19:32:45.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.456+0000 7f75d0cd9640 1 -- 172.21.15.19:0/935470727 wait complete. 2024-03-25T19:32:45.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.456+0000 7f75d0cd9640 10 monclient: build_initial_monmap 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.456+0000 7f75d0cd9640 10 monclient: monmap: 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.457+0000 7f75d0cd9640 1 librados: starting msgr at 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.457+0000 7f75d0cd9640 1 librados: starting objecter 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.457+0000 7f75d0cd9640 1 Processor -- start 2024-03-25T19:32:45.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.457+0000 7f75d0cd9640 1 -- start start 2024-03-25T19:32:45.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.457+0000 7f75d0cd9640 1 librados: setting wanted keys 2024-03-25T19:32:45.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.457+0000 7f75d0cd9640 1 librados: calling monclient init 2024-03-25T19:32:45.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.457+0000 7f75d0cd9640 10 monclient: init 2024-03-25T19:32:45.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.458+0000 7f75d0cd9640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:45.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.458+0000 7f75d0cd9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:45.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.458+0000 7f75d0cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f75cc06ab80 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:45.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.458+0000 7f75d0cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f75cc0b00b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:45.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.458+0000 7f75d0cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f75cc145d90 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:45.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.458+0000 7f75d0cd9640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc13cc20 con 0x7f75cc145d90 2024-03-25T19:32:45.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.458+0000 7f75d0cd9640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc136fd0 con 0x7f75cc06ab80 2024-03-25T19:32:45.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.458+0000 7f75d0cd9640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc137160 con 0x7f75cc0b00b0 2024-03-25T19:32:45.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.458+0000 7f75d0cd9640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:45.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.458+0000 7f75d0cd9640 10 monclient(hunting): authenticate will time out at 6218.728228s 2024-03-25T19:32:45.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.460+0000 7f75ca59c640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f75cc06ab80 0x7f75cc1db230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:56586/0 (socket says 172.21.15.19:56586) 2024-03-25T19:32:45.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.460+0000 7f75ca59c640 1 -- 172.21.15.19:0/694177627 learned_addr learned my addr 172.21.15.19:0/694177627 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:45.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.461+0000 7f75c2ffd640 10 client.?.objecter ms_handle_connect 0x7f75cc06ab80 2024-03-25T19:32:45.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.461+0000 7f75c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:45.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.464+0000 7f75c2ffd640 10 client.?.objecter ms_handle_connect 0x7f75cc0b00b0 2024-03-25T19:32:45.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.464+0000 7f75c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:45.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.471+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794121864 0 0) 0x7f75cc136fd0 con 0x7f75cc06ab80 2024-03-25T19:32:45.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.471+0000 7f75c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:45.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.471+0000 7f75c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:45.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.471+0000 7f75c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:45.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.471+0000 7f75c2ffd640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:32:45.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.471+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75ac004eb0 con 0x7f75cc06ab80 2024-03-25T19:32:45.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.471+0000 7f75c2ffd640 10 client.?.objecter ms_handle_connect 0x7f75cc145d90 2024-03-25T19:32:45.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.471+0000 7f75c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:45.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.474+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438569630 0 0) 0x7f75cc137160 con 0x7f75cc0b00b0 2024-03-25T19:32:45.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.474+0000 7f75c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:45.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.474+0000 7f75c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:45.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.474+0000 7f75c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:45.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.474+0000 7f75c2ffd640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:32:45.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.474+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75cc136fd0 con 0x7f75cc0b00b0 2024-03-25T19:32:45.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.481+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824789796 0 0) 0x7f75cc13cc20 con 0x7f75cc145d90 2024-03-25T19:32:45.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.481+0000 7f75c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:45.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.481+0000 7f75c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:45.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.481+0000 7f75c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:45.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.481+0000 7f75c2ffd640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:32:45.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.481+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75cc137160 con 0x7f75cc145d90 2024-03-25T19:32:45.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.489+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 809524178 0 0) 0x7f75ac004eb0 con 0x7f75cc06ab80 2024-03-25T19:32:45.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.489+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75cc13cc20 con 0x7f75cc06ab80 2024-03-25T19:32:45.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.489+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f75bc001a10 con 0x7f75cc06ab80 2024-03-25T19:32:45.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.489+0000 7f75c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.489+0000 7f75c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.489+0000 7f75c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:45.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.489+0000 7f75c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:45.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.492+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3997076313 0 0) 0x7f75cc136fd0 con 0x7f75cc0b00b0 2024-03-25T19:32:45.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.492+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75ac004eb0 con 0x7f75cc0b00b0 2024-03-25T19:32:45.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.492+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f75b40018b0 con 0x7f75cc0b00b0 2024-03-25T19:32:45.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.492+0000 7f75c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:45.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.492+0000 7f75c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:45.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.492+0000 7f75c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:32:45.492 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:45.492 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:45.492 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:45.492 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:45.492 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:45.492 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:45.493 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:45.493 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:45.493 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:45.493 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:45.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1914625217 0 0) 0x7f75cc13cc20 con 0x7f75cc06ab80 2024-03-25T19:32:45.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 >> v1:172.21.15.19:6790/0 conn(0x7f75cc0b00b0 legacy=0x7f75cc1db940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:45.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 >> v1:172.21.15.19:6789/0 conn(0x7f75cc145d90 legacy=0x7f75cc1e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:45.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:45.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75c2ffd640 1 monclient: found mon.b 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75c2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc1386b0 con 0x7f75cc06ab80 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75c2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75d0cd9640 5 monclient: authenticate success, global_id 4390 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75d0cd9640 10 monclient: _renew_subs 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75d0cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75d0cd9640 1 -- 172.21.15.19:0/694177627 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc138920 con 0x7f75cc06ab80 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75d0cd9640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75d0cd9640 10 monclient: _renew_subs 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75d0cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:45.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75d0cd9640 1 -- 172.21.15.19:0/694177627 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc1e3e00 con 0x7f75cc06ab80 2024-03-25T19:32:45.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.504+0000 7f75d0cd9640 1 librados: init done 2024-03-25T19:32:45.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.505+0000 7f75d0cd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:45.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.505+0000 7f75d0cd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:45.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.505+0000 7f75d0cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:45.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.505+0000 7f75d0cd9640 1 -- 172.21.15.19:0/694177627 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75cc136e20 con 0x7f75cc06ab80 2024-03-25T19:32:45.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.776+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75bc001c50 con 0x7f75cc06ab80 2024-03-25T19:32:45.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.776+0000 7f75c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:45.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.776+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f75bc003a60 con 0x7f75cc06ab80 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.776+0000 7f75c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.776+0000 7f75c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.776+0000 7f75c2ffd640 10 monclient: dump: 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:45.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.776+0000 7f75c959a640 4 set_mon_vals no callback set 2024-03-25T19:32:45.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.777+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f75bc003c80 con 0x7f75cc06ab80 2024-03-25T19:32:45.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.778+0000 7f75c3fff640 10 monclient: get_auth_request con 0x7f75ac06df70 auth_method 0 2024-03-25T19:32:45.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.778+0000 7f75c2ffd640 10 client.4390.objecter ms_handle_connect 0x7f75ac06df70 2024-03-25T19:32:45.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.781+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f75bc054880 con 0x7f75cc06ab80 2024-03-25T19:32:45.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.781+0000 7f75c2ffd640 10 client.4390.objecter ms_dispatch 0x7f75cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:45.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.781+0000 7f75c2ffd640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:45.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.781+0000 7f75c2ffd640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:45.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.781+0000 7f75c2ffd640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-25T19:32:45.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.781+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75bc050390 con 0x7f75cc06ab80 2024-03-25T19:32:45.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.781+0000 7f75c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:45.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:45.781+0000 7f75c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:46.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.032+0000 7f75d0cd9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:46.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.032+0000 7f75d0cd9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:46.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.032+0000 7f75d0cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:46.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.032+0000 7f75d0cd9640 1 -- 172.21.15.19:0/694177627 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75cc138010 con 0x7f75cc06ab80 2024-03-25T19:32:46.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.112+0000 7f75c2ffd640 1 -- 172.21.15.19:0/694177627 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f75cc138010 con 0x7f75cc06ab80 2024-03-25T19:32:46.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.113+0000 7f75c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:46.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.113+0000 7f75c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:46.113 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:32:46.113 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 10 librados: watch_flush enter 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 10 librados: watch_flush exit 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 10 client.4390.objecter successfully canceled tick 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 1 -- 172.21.15.19:0/694177627 >> v1:172.21.15.188:6800/2207506229 conn(0x7f75ac06df70 legacy=0x7f75ac070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 10 monclient: shutdown 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 1 -- 172.21.15.19:0/694177627 >> v1:172.21.15.188:6789/0 conn(0x7f75cc06ab80 legacy=0x7f75cc1db230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 1 -- 172.21.15.19:0/694177627 shutdown_connections 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 1 -- 172.21.15.19:0/694177627 >> 172.21.15.19:0/694177627 conn(0x7f75cc0b4a80 msgr2=0x7f75cc0b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 1 -- 172.21.15.19:0/694177627 shutdown_connections 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 1 -- 172.21.15.19:0/694177627 wait complete. 2024-03-25T19:32:46.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:46.115+0000 7f75d0cd9640 1 librados: shutdown 2024-03-25T19:32:46.126 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:52.933 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:53.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.046+0000 7feda5fc8640 10 monclient: get_monmap_and_config 2024-03-25T19:32:53.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.046+0000 7feda5fc8640 10 monclient: build_initial_monmap 2024-03-25T19:32:53.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.046+0000 7feda5fc8640 10 monclient: monmap: 2024-03-25T19:32:53.047 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:53.047 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:53.047 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:32:53.047955+0000 2024-03-25T19:32:53.047 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:32:53.047955+0000 2024-03-25T19:32:53.047 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:53.047 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:53.047 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:53.047 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:32:53.047 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:32:53.047 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:53.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.047+0000 7feda5fc8640 1 Processor -- start 2024-03-25T19:32:53.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.047+0000 7feda5fc8640 1 -- start start 2024-03-25T19:32:53.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.047+0000 7feda5fc8640 10 monclient: init 2024-03-25T19:32:53.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.048+0000 7feda5fc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:53.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.048+0000 7feda5fc8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:32:53.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.048+0000 7feda5fc8640 10 monclient(hunting): picked mon.noname-a con 0x7feda00a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:53.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.048+0000 7feda5fc8640 10 monclient(hunting): picked mon.noname-b con 0x7feda013f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:53.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.048+0000 7feda5fc8640 10 monclient(hunting): picked mon.noname-c con 0x7feda01430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:53.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.048+0000 7feda5fc8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda0142520 con 0x7feda00a98f0 2024-03-25T19:32:53.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.048+0000 7feda5fc8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda00b28a0 con 0x7feda01430c0 2024-03-25T19:32:53.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.048+0000 7feda5fc8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda00b2a30 con 0x7feda013f710 2024-03-25T19:32:53.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.048+0000 7feda5fc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:53.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.048+0000 7feda5fc8640 10 monclient(hunting): authenticate will time out at 6226.318780s 2024-03-25T19:32:53.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.053+0000 7fed9effd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7feda013f710 0x7feda01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:52110/0 (socket says 172.21.15.19:52110) 2024-03-25T19:32:53.053 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.053+0000 7fed9effd640 1 -- 172.21.15.19:0/479024559 learned_addr learned my addr 172.21.15.19:0/479024559 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:53.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.064+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460063115 0 0) 0x7feda00b2a30 con 0x7feda013f710 2024-03-25T19:32:53.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.064+0000 7fed9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:53.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.064+0000 7fed9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:53.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.064+0000 7fed9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:53.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.064+0000 7fed9e7fc640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:32:53.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.064+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed88003520 con 0x7feda013f710 2024-03-25T19:32:53.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.071+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117828539 0 0) 0x7feda00b28a0 con 0x7feda01430c0 2024-03-25T19:32:53.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.071+0000 7fed9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:53.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.071+0000 7fed9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:53.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.071+0000 7fed9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:53.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.071+0000 7fed9e7fc640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:32:53.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.071+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda00b2a30 con 0x7feda01430c0 2024-03-25T19:32:53.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.072+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021543549 0 0) 0x7feda0142520 con 0x7feda00a98f0 2024-03-25T19:32:53.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.072+0000 7fed9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:53.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.072+0000 7fed9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:53.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.072+0000 7fed9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:53.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.072+0000 7fed9e7fc640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:32:53.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.072+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda00b28a0 con 0x7feda00a98f0 2024-03-25T19:32:53.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3242026036 0 0) 0x7fed88003520 con 0x7feda013f710 2024-03-25T19:32:53.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 >> v1:172.21.15.188:6789/0 conn(0x7feda01430c0 legacy=0x7feda014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 >> v1:172.21.15.19:6789/0 conn(0x7feda00a98f0 legacy=0x7feda00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda00b44c0 con 0x7feda013f710 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fed94001690 con 0x7feda013f710 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7feda5fc8640 5 monclient: authenticate success, global_id 4373 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7feda5fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:53.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7fed9e7fc640 10 monclient: dump: 2024-03-25T19:32:53.098 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:53.098 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:53.098 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:53.098 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:53.098 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:53.098 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:53.098 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:53.098 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:53.098 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:53.098 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:53.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.096+0000 7feda5fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:53.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.105+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed940017f0 con 0x7feda013f710 2024-03-25T19:32:53.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.105+0000 7fed9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:53.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.105+0000 7fed9e7fc640 1 -- 172.21.15.19:0/479024559 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fed94001c00 con 0x7feda013f710 2024-03-25T19:32:53.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.105+0000 7fed9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:53.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.105+0000 7fed9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:53.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.105+0000 7fed9e7fc640 10 monclient: dump: 2024-03-25T19:32:53.106 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:53.106 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:53.106 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:53.106 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:53.106 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:53.106 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:53.106 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:53.106 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:53.106 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:53.106 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:53.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.105+0000 7feda5fc8640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:53.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.105+0000 7feda5fc8640 4 set_mon_vals no callback set 2024-03-25T19:32:53.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.106+0000 7feda5fc8640 10 monclient: shutdown 2024-03-25T19:32:53.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.106+0000 7feda5fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.106+0000 7feda5fc8640 1 -- 172.21.15.19:0/479024559 >> v1:172.21.15.19:6790/0 conn(0x7feda013f710 legacy=0x7feda01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.106+0000 7feda5fc8640 1 -- 172.21.15.19:0/479024559 shutdown_connections 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.106+0000 7feda5fc8640 1 -- 172.21.15.19:0/479024559 >> 172.21.15.19:0/479024559 conn(0x7feda00b7ea0 msgr2=0x7feda00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.106+0000 7feda5fc8640 1 -- 172.21.15.19:0/479024559 shutdown_connections 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.106+0000 7feda5fc8640 1 -- 172.21.15.19:0/479024559 wait complete. 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.107+0000 7feda5fc8640 10 monclient: build_initial_monmap 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.107+0000 7feda5fc8640 10 monclient: monmap: 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:32:53.108 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:32:53.109 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:32:53.109 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:53.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.108+0000 7feda5fc8640 1 librados: starting msgr at 2024-03-25T19:32:53.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.108+0000 7feda5fc8640 1 librados: starting objecter 2024-03-25T19:32:53.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.108+0000 7feda5fc8640 1 Processor -- start 2024-03-25T19:32:53.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.108+0000 7feda5fc8640 1 -- start start 2024-03-25T19:32:53.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.108+0000 7feda5fc8640 1 librados: setting wanted keys 2024-03-25T19:32:53.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.108+0000 7feda5fc8640 1 librados: calling monclient init 2024-03-25T19:32:53.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.108+0000 7feda5fc8640 10 monclient: init 2024-03-25T19:32:53.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.108+0000 7feda5fc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:53.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.108+0000 7feda5fc8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:53.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.108+0000 7feda5fc8640 10 monclient(hunting): picked mon.noname-b con 0x7feda00a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:32:53.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.109+0000 7feda5fc8640 10 monclient(hunting): picked mon.noname-c con 0x7feda013f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:32:53.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.109+0000 7feda5fc8640 10 monclient(hunting): picked mon.noname-a con 0x7feda01430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:32:53.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.109+0000 7feda5fc8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda00ab8b0 con 0x7feda01430c0 2024-03-25T19:32:53.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.109+0000 7feda5fc8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda00abb90 con 0x7feda00a98f0 2024-03-25T19:32:53.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.109+0000 7feda5fc8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda0137870 con 0x7feda013f710 2024-03-25T19:32:53.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.109+0000 7feda5fc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:53.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.109+0000 7feda5fc8640 10 monclient(hunting): authenticate will time out at 6226.379035s 2024-03-25T19:32:53.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.122+0000 7fed9effd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7feda013f710 0x7feda01df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:52164/0 (socket says 172.21.15.19:52164) 2024-03-25T19:32:53.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.122+0000 7fed9effd640 1 -- 172.21.15.19:0/1999028451 learned_addr learned my addr 172.21.15.19:0/1999028451 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:32:53.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.124+0000 7fed87fff640 10 client.?.objecter ms_handle_connect 0x7feda013f710 2024-03-25T19:32:53.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.124+0000 7fed87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:53.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.125+0000 7fed87fff640 10 client.?.objecter ms_handle_connect 0x7feda00a98f0 2024-03-25T19:32:53.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.125+0000 7fed87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:53.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.128+0000 7fed87fff640 10 client.?.objecter ms_handle_connect 0x7feda01430c0 2024-03-25T19:32:53.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.128+0000 7fed87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:53.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.130+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848800297 0 0) 0x7feda0137870 con 0x7feda013f710 2024-03-25T19:32:53.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.130+0000 7fed87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:53.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.130+0000 7fed87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:53.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.130+0000 7fed87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:53.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.130+0000 7fed87fff640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:32:53.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.130+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed8c004eb0 con 0x7feda013f710 2024-03-25T19:32:53.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.137+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092124888 0 0) 0x7feda00abb90 con 0x7feda00a98f0 2024-03-25T19:32:53.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.137+0000 7fed87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:53.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.137+0000 7fed87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:53.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.137+0000 7fed87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:53.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.137+0000 7fed87fff640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:32:53.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.137+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda0137870 con 0x7feda00a98f0 2024-03-25T19:32:53.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.144+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3576126927 0 0) 0x7fed8c004eb0 con 0x7feda013f710 2024-03-25T19:32:53.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.144+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feda00abb90 con 0x7feda013f710 2024-03-25T19:32:53.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.144+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fed94001bc0 con 0x7feda013f710 2024-03-25T19:32:53.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.144+0000 7fed87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:53.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.144+0000 7fed87fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:53.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.144+0000 7fed87fff640 10 monclient(hunting): dump: 2024-03-25T19:32:53.145 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:53.145 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:53.145 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:53.145 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:53.145 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:53.145 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:53.145 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:53.145 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:53.145 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:53.145 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:53.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.144+0000 7fed87fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:53.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.148+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006028987 0 0) 0x7feda00ab8b0 con 0x7feda01430c0 2024-03-25T19:32:53.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.148+0000 7fed87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:53.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.148+0000 7fed87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:53.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.148+0000 7fed87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:53.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.148+0000 7fed87fff640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:32:53.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.148+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed8c004eb0 con 0x7feda01430c0 2024-03-25T19:32:53.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.153+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 95029858 0 0) 0x7feda0137870 con 0x7feda00a98f0 2024-03-25T19:32:53.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.153+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feda00ab8b0 con 0x7feda00a98f0 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.153+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fed90001870 con 0x7feda00a98f0 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.153+0000 7fed87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.153+0000 7fed87fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.153+0000 7fed87fff640 10 monclient(hunting): dump: 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:53.154 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:53.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1988570757 0 0) 0x7feda00ab8b0 con 0x7feda00a98f0 2024-03-25T19:32:53.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 >> v1:172.21.15.19:6790/0 conn(0x7feda013f710 legacy=0x7feda01df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:53.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 >> v1:172.21.15.19:6789/0 conn(0x7feda01430c0 legacy=0x7feda01e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:53.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7fed87fff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:53.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7fed87fff640 1 monclient: found mon.b 2024-03-25T19:32:53.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7fed87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:53.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7fed87fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda01389e0 con 0x7feda00a98f0 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7fed87fff640 10 monclient: _finish_auth 0 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7fed87fff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7fed87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7feda5fc8640 5 monclient: authenticate success, global_id 4396 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7feda5fc8640 10 monclient: _renew_subs 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7feda5fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7feda5fc8640 1 -- 172.21.15.19:0/1999028451 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda0136d90 con 0x7feda00a98f0 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7feda5fc8640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7feda5fc8640 10 monclient: _renew_subs 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7feda5fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7feda5fc8640 1 -- 172.21.15.19:0/1999028451 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feda01384c0 con 0x7feda00a98f0 2024-03-25T19:32:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.165+0000 7feda5fc8640 1 librados: init done 2024-03-25T19:32:53.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.166+0000 7feda5fc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:53.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.166+0000 7feda5fc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:53.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.166+0000 7feda5fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:53.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.166+0000 7feda5fc8640 1 -- 172.21.15.19:0/1999028451 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda0136f80 con 0x7feda00a98f0 2024-03-25T19:32:53.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.447+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed90003610 con 0x7feda00a98f0 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.447+0000 7fed87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.447+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fed90001d80 con 0x7feda00a98f0 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.447+0000 7fed87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.447+0000 7fed87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.447+0000 7fed87fff640 10 monclient: dump: 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:32:53.448 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:32:53.449 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:32:53.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.447+0000 7fed9dffb640 4 set_mon_vals no callback set 2024-03-25T19:32:53.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.448+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fed90003b20 con 0x7feda00a98f0 2024-03-25T19:32:53.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.449+0000 7fed9effd640 10 monclient: get_auth_request con 0x7fed8c06df50 auth_method 0 2024-03-25T19:32:53.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.449+0000 7fed87fff640 10 client.4396.objecter ms_handle_connect 0x7fed8c06df50 2024-03-25T19:32:53.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.453+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fed90053590 con 0x7feda00a98f0 2024-03-25T19:32:53.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.453+0000 7fed87fff640 10 client.4396.objecter ms_dispatch 0x7feda0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:53.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.453+0000 7fed87fff640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:53.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.453+0000 7fed87fff640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:53.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.453+0000 7fed87fff640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T19:32:53.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.453+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed9004f0a0 con 0x7feda00a98f0 2024-03-25T19:32:53.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.454+0000 7fed87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:53.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.454+0000 7fed87fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:53.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.673+0000 7feda5fc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:53.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.673+0000 7feda5fc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:53.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.673+0000 7feda5fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:32:53.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.673+0000 7feda5fc8640 1 -- 172.21.15.19:0/1999028451 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feda00627b0 con 0x7feda00a98f0 2024-03-25T19:32:53.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.710+0000 7fed87fff640 1 -- 172.21.15.19:0/1999028451 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fed90050fd0 con 0x7feda00a98f0 2024-03-25T19:32:53.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.710+0000 7fed87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:53.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.710+0000 7fed87fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:53.710 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:32:53.710 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:53.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 10 librados: watch_flush enter 2024-03-25T19:32:53.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 10 librados: watch_flush exit 2024-03-25T19:32:53.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T19:32:53.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 10 client.4396.objecter successfully canceled tick 2024-03-25T19:32:53.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 1 -- 172.21.15.19:0/1999028451 >> v1:172.21.15.188:6800/2207506229 conn(0x7fed8c06df50 legacy=0x7fed8c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:53.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 10 monclient: shutdown 2024-03-25T19:32:53.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:53.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 1 -- 172.21.15.19:0/1999028451 >> v1:172.21.15.188:6789/0 conn(0x7feda00a98f0 legacy=0x7feda01350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:53.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 1 -- 172.21.15.19:0/1999028451 shutdown_connections 2024-03-25T19:32:53.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 1 -- 172.21.15.19:0/1999028451 >> 172.21.15.19:0/1999028451 conn(0x7feda00b7ea0 msgr2=0x7feda0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:53.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 1 -- 172.21.15.19:0/1999028451 shutdown_connections 2024-03-25T19:32:53.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 1 -- 172.21.15.19:0/1999028451 wait complete. 2024-03-25T19:32:53.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:32:53.712+0000 7feda5fc8640 1 librados: shutdown 2024-03-25T19:32:53.723 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:00.530 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:00.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.647+0000 7f703675d640 10 monclient: get_monmap_and_config 2024-03-25T19:33:00.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.647+0000 7f703675d640 10 monclient: build_initial_monmap 2024-03-25T19:33:00.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.647+0000 7f703675d640 10 monclient: monmap: 2024-03-25T19:33:00.648 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:00.648 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:00.648 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:33:00.649013+0000 2024-03-25T19:33:00.648 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:33:00.649013+0000 2024-03-25T19:33:00.648 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:00.648 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:00.648 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:00.648 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:33:00.648 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:33:00.648 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:00.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.648+0000 7f703675d640 1 Processor -- start 2024-03-25T19:33:00.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.648+0000 7f703675d640 1 -- start start 2024-03-25T19:33:00.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.648+0000 7f703675d640 10 monclient: init 2024-03-25T19:33:00.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.649+0000 7f703675d640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:00.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.649+0000 7f703675d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:00.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.649+0000 7f703675d640 10 monclient(hunting): picked mon.noname-c con 0x7f70300b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:00.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.649+0000 7f703675d640 10 monclient(hunting): picked mon.noname-b con 0x7f70300b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:00.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.650+0000 7f703675d640 10 monclient(hunting): picked mon.noname-a con 0x7f703014be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:00.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.650+0000 7f703675d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70300af940 con 0x7f703014be90 2024-03-25T19:33:00.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.650+0000 7f703675d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70300b0a70 con 0x7f70300b4d60 2024-03-25T19:33:00.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.650+0000 7f703675d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70300b0d50 con 0x7f70300b1730 2024-03-25T19:33:00.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.650+0000 7f703675d640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:00.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.650+0000 7f703675d640 10 monclient(hunting): authenticate will time out at 6233.920349s 2024-03-25T19:33:00.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.652+0000 7f702ffff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f70300b4d60 0x7f703013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:38794/0 (socket says 172.21.15.19:38794) 2024-03-25T19:33:00.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.652+0000 7f702ffff640 1 -- 172.21.15.19:0/498383520 learned_addr learned my addr 172.21.15.19:0/498383520 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:00.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.660+0000 7f702effd640 1 -- 172.21.15.19:0/498383520 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204333318 0 0) 0x7f70300b0a70 con 0x7f70300b4d60 2024-03-25T19:33:00.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.660+0000 7f702effd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:00.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.660+0000 7f702effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:00.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.660+0000 7f702effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:00.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.660+0000 7f702effd640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:33:00.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.660+0000 7f702effd640 1 -- 172.21.15.19:0/498383520 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7018003520 con 0x7f70300b4d60 2024-03-25T19:33:00.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.674+0000 7f702effd640 1 -- 172.21.15.19:0/498383520 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3558429851 0 0) 0x7f7018003520 con 0x7f70300b4d60 2024-03-25T19:33:00.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 1 -- 172.21.15.19:0/498383520 >> v1:172.21.15.19:6790/0 conn(0x7f70300b1730 legacy=0x7f70300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 1 -- 172.21.15.19:0/498383520 >> v1:172.21.15.19:6789/0 conn(0x7f703014be90 legacy=0x7f703014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:00.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 1 monclient: found mon.noname-c 2024-03-25T19:33:00.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:00.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 1 -- 172.21.15.19:0/498383520 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70300b1030 con 0x7f70300b4d60 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 10 monclient: _finish_auth 0 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 1 -- 172.21.15.19:0/498383520 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7020001690 con 0x7f70300b4d60 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f703675d640 5 monclient: authenticate success, global_id 4399 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f703675d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f702effd640 10 monclient: dump: 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:00.676 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:00.677 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:00.677 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:00.677 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:00.677 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:00.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.675+0000 7f703675d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:00.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.701+0000 7f702effd640 1 -- 172.21.15.19:0/498383520 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70200017f0 con 0x7f70300b4d60 2024-03-25T19:33:00.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.701+0000 7f702effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:00.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.701+0000 7f702effd640 1 -- 172.21.15.19:0/498383520 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7020001c40 con 0x7f70300b4d60 2024-03-25T19:33:00.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.701+0000 7f702effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:00.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.701+0000 7f702effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:00.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.701+0000 7f702effd640 10 monclient: dump: 2024-03-25T19:33:00.702 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:00.702 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:00.702 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:00.702 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:00.702 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:00.702 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:00.702 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:00.703 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:00.703 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:00.703 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:00.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.701+0000 7f703675d640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:00.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.701+0000 7f703675d640 4 set_mon_vals no callback set 2024-03-25T19:33:00.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.702+0000 7f703675d640 10 monclient: shutdown 2024-03-25T19:33:00.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.702+0000 7f703675d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:00.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.702+0000 7f703675d640 1 -- 172.21.15.19:0/498383520 >> v1:172.21.15.188:6789/0 conn(0x7f70300b4d60 legacy=0x7f703013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.702+0000 7f703675d640 1 -- 172.21.15.19:0/498383520 shutdown_connections 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.702+0000 7f703675d640 1 -- 172.21.15.19:0/498383520 >> 172.21.15.19:0/498383520 conn(0x7f70300617a0 msgr2=0x7f7030061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.702+0000 7f703675d640 1 -- 172.21.15.19:0/498383520 shutdown_connections 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.703+0000 7f703675d640 1 -- 172.21.15.19:0/498383520 wait complete. 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.703+0000 7f703675d640 10 monclient: build_initial_monmap 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.703+0000 7f703675d640 10 monclient: monmap: 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:33:00.705 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:00.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.704+0000 7f703675d640 1 librados: starting msgr at 2024-03-25T19:33:00.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.704+0000 7f703675d640 1 librados: starting objecter 2024-03-25T19:33:00.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.704+0000 7f703675d640 1 Processor -- start 2024-03-25T19:33:00.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.704+0000 7f703675d640 1 -- start start 2024-03-25T19:33:00.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.704+0000 7f703675d640 1 librados: setting wanted keys 2024-03-25T19:33:00.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.704+0000 7f703675d640 1 librados: calling monclient init 2024-03-25T19:33:00.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.704+0000 7f703675d640 10 monclient: init 2024-03-25T19:33:00.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.705+0000 7f703675d640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:00.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.705+0000 7f703675d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:00.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.705+0000 7f703675d640 10 monclient(hunting): picked mon.noname-b con 0x7f70300b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:00.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.705+0000 7f703675d640 10 monclient(hunting): picked mon.noname-c con 0x7f70300b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:00.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.705+0000 7f703675d640 10 monclient(hunting): picked mon.noname-a con 0x7f703014be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:00.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.705+0000 7f703675d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7030141f40 con 0x7f703014be90 2024-03-25T19:33:00.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.705+0000 7f703675d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7030142360 con 0x7f70300b1730 2024-03-25T19:33:00.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.705+0000 7f703675d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70301424f0 con 0x7f70300b4d60 2024-03-25T19:33:00.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.705+0000 7f703675d640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:00.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.705+0000 7f703675d640 10 monclient(hunting): authenticate will time out at 6233.975436s 2024-03-25T19:33:00.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.719+0000 7f7034cfa640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f703014be90 0x7f70301e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:37344/0 (socket says 172.21.15.19:37344) 2024-03-25T19:33:00.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.719+0000 7f7034cfa640 1 -- 172.21.15.19:0/1873067363 learned_addr learned my addr 172.21.15.19:0/1873067363 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:00.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.721+0000 7f702cff9640 10 client.?.objecter ms_handle_connect 0x7f70300b1730 2024-03-25T19:33:00.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.721+0000 7f702cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:00.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.724+0000 7f702cff9640 10 client.?.objecter ms_handle_connect 0x7f703014be90 2024-03-25T19:33:00.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.724+0000 7f702cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:00.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.728+0000 7f702cff9640 10 client.?.objecter ms_handle_connect 0x7f70300b4d60 2024-03-25T19:33:00.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.728+0000 7f702cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:00.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.735+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145664267 0 0) 0x7f7030141f40 con 0x7f703014be90 2024-03-25T19:33:00.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.735+0000 7f702cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:00.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.735+0000 7f702cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:00.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.735+0000 7f702cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:00.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.735+0000 7f702cff9640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:33:00.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.735+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f701c004eb0 con 0x7f703014be90 2024-03-25T19:33:00.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.822+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 561584960 0 0) 0x7f701c004eb0 con 0x7f703014be90 2024-03-25T19:33:00.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.822+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7030141f40 con 0x7f703014be90 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.822+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7028001c90 con 0x7f703014be90 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.822+0000 7f702cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.822+0000 7f702cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.822+0000 7f702cff9640 10 monclient(hunting): dump: 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:00.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.822+0000 7f702cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:00.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2685315357 0 0) 0x7f7030141f40 con 0x7f703014be90 2024-03-25T19:33:00.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 >> v1:172.21.15.19:6790/0 conn(0x7f70300b4d60 legacy=0x7f70301d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 >> v1:172.21.15.188:6789/0 conn(0x7f70300b1730 legacy=0x7f70301405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f702cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:33:00.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f702cff9640 1 monclient: found mon.a 2024-03-25T19:33:00.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f702cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:00.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f702cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7030140d00 con 0x7f703014be90 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f702cff9640 10 monclient: _finish_auth 0 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f702cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f702cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f703675d640 5 monclient: authenticate success, global_id 4410 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f703675d640 10 monclient: _renew_subs 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f703675d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.846+0000 7f703675d640 1 -- 172.21.15.19:0/1873067363 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7030138150 con 0x7f703014be90 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.847+0000 7f703675d640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.847+0000 7f703675d640 10 monclient: _renew_subs 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.847+0000 7f703675d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.847+0000 7f703675d640 1 -- 172.21.15.19:0/1873067363 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70301d8520 con 0x7f703014be90 2024-03-25T19:33:00.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.847+0000 7f703675d640 1 librados: init done 2024-03-25T19:33:00.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.848+0000 7f703675d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:00.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.848+0000 7f703675d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:00.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.848+0000 7f703675d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:00.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:00.848+0000 7f703675d640 1 -- 172.21.15.19:0/1873067363 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7030140f50 con 0x7f703014be90 2024-03-25T19:33:01.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.156+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7028003990 con 0x7f703014be90 2024-03-25T19:33:01.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.156+0000 7f702cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:01.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.156+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7028003da0 con 0x7f703014be90 2024-03-25T19:33:01.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.156+0000 7f702cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:01.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.156+0000 7f702cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:01.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.156+0000 7f702cff9640 10 monclient: dump: 2024-03-25T19:33:01.157 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:01.157 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:01.157 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:01.157 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:01.157 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:01.157 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:01.157 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:01.157 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:01.157 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:01.157 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:01.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.156+0000 7f702e7fc640 4 set_mon_vals no callback set 2024-03-25T19:33:01.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.157+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f7028003fc0 con 0x7f703014be90 2024-03-25T19:33:01.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.157+0000 7f702ffff640 10 monclient: get_auth_request con 0x7f701c06dd90 auth_method 0 2024-03-25T19:33:01.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.157+0000 7f702cff9640 10 client.4410.objecter ms_handle_connect 0x7f701c06dd90 2024-03-25T19:33:01.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.163+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f7028054aa0 con 0x7f703014be90 2024-03-25T19:33:01.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.163+0000 7f702cff9640 10 client.4410.objecter ms_dispatch 0x7f7030000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:01.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.163+0000 7f702cff9640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:01.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.163+0000 7f702cff9640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:01.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.163+0000 7f702cff9640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-25T19:33:01.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.163+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70280505b0 con 0x7f703014be90 2024-03-25T19:33:01.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.163+0000 7f702cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:01.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.163+0000 7f702cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:01.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.430+0000 7f703675d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:01.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.430+0000 7f703675d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:01.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.430+0000 7f703675d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:01.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.430+0000 7f703675d640 1 -- 172.21.15.19:0/1873067363 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70301d7950 con 0x7f703014be90 2024-03-25T19:33:01.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.475+0000 7f702cff9640 1 -- 172.21.15.19:0/1873067363 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f70301d7950 con 0x7f703014be90 2024-03-25T19:33:01.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.475+0000 7f702cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:01.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.475+0000 7f702cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:01.475 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:33:01.475 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:01.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 10 librados: watch_flush enter 2024-03-25T19:33:01.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 10 librados: watch_flush exit 2024-03-25T19:33:01.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-25T19:33:01.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 10 client.4410.objecter successfully canceled tick 2024-03-25T19:33:01.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 1 -- 172.21.15.19:0/1873067363 >> v1:172.21.15.188:6800/2207506229 conn(0x7f701c06dd90 legacy=0x7f701c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:01.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 10 monclient: shutdown 2024-03-25T19:33:01.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:01.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 1 -- 172.21.15.19:0/1873067363 >> v1:172.21.15.19:6789/0 conn(0x7f703014be90 legacy=0x7f70301e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:01.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 1 -- 172.21.15.19:0/1873067363 shutdown_connections 2024-03-25T19:33:01.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 1 -- 172.21.15.19:0/1873067363 >> 172.21.15.19:0/1873067363 conn(0x7f70300617a0 msgr2=0x7f70300b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:01.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 1 -- 172.21.15.19:0/1873067363 shutdown_connections 2024-03-25T19:33:01.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 1 -- 172.21.15.19:0/1873067363 wait complete. 2024-03-25T19:33:01.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:01.477+0000 7f703675d640 1 librados: shutdown 2024-03-25T19:33:01.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:08.294 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:08.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.413+0000 7f2afd086640 10 monclient: get_monmap_and_config 2024-03-25T19:33:08.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.413+0000 7f2afd086640 10 monclient: build_initial_monmap 2024-03-25T19:33:08.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.413+0000 7f2afd086640 10 monclient: monmap: 2024-03-25T19:33:08.414 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:08.414 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:08.414 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:33:08.415003+0000 2024-03-25T19:33:08.414 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:33:08.415003+0000 2024-03-25T19:33:08.414 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:08.414 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:08.414 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:08.414 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:33:08.414 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:33:08.414 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.414+0000 7f2afd086640 1 Processor -- start 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.414+0000 7f2afd086640 1 -- start start 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.414+0000 7f2afd086640 10 monclient: init 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.415+0000 7f2afd086640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.415+0000 7f2afd086640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.415+0000 7f2afd086640 10 monclient(hunting): picked mon.noname-a con 0x7f2af80b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.415+0000 7f2afd086640 10 monclient(hunting): picked mon.noname-c con 0x7f2af80b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.415+0000 7f2afd086640 10 monclient(hunting): picked mon.noname-b con 0x7f2af814be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.415+0000 7f2afd086640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af80af940 con 0x7f2af80b4d60 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.415+0000 7f2afd086640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af80b0a70 con 0x7f2af80b1730 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.415+0000 7f2afd086640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af80b0d50 con 0x7f2af814be90 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.415+0000 7f2afd086640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:08.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.415+0000 7f2afd086640 10 monclient(hunting): authenticate will time out at 6241.685499s 2024-03-25T19:33:08.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.422+0000 7f2af6d9d640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f2af80b4d60 0x7f2af813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:42124/0 (socket says 172.21.15.19:42124) 2024-03-25T19:33:08.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.422+0000 7f2af6d9d640 1 -- 172.21.15.19:0/2868085662 learned_addr learned my addr 172.21.15.19:0/2868085662 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:08.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.439+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364500498 0 0) 0x7f2af80b0d50 con 0x7f2af814be90 2024-03-25T19:33:08.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.439+0000 7f2af659c640 10 monclient(hunting): _negotiate 2024-03-25T19:33:08.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.439+0000 7f2af659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:08.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.439+0000 7f2af659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:08.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.439+0000 7f2af659c640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:33:08.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.439+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae4003520 con 0x7f2af814be90 2024-03-25T19:33:08.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.443+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288315554 0 0) 0x7f2af80af940 con 0x7f2af80b4d60 2024-03-25T19:33:08.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.443+0000 7f2af659c640 10 monclient(hunting): _negotiate 2024-03-25T19:33:08.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.443+0000 7f2af659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:08.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.443+0000 7f2af659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:08.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.443+0000 7f2af659c640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:33:08.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.443+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af80b0d50 con 0x7f2af80b4d60 2024-03-25T19:33:08.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.447+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252683294 0 0) 0x7f2af80b0a70 con 0x7f2af80b1730 2024-03-25T19:33:08.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.447+0000 7f2af659c640 10 monclient(hunting): _negotiate 2024-03-25T19:33:08.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.447+0000 7f2af659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:08.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.447+0000 7f2af659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:08.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.447+0000 7f2af659c640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:33:08.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.447+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af80af940 con 0x7f2af80b1730 2024-03-25T19:33:08.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3958834438 0 0) 0x7f2ae4003520 con 0x7f2af814be90 2024-03-25T19:33:08.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 >> v1:172.21.15.188:6789/0 conn(0x7f2af80b1730 legacy=0x7f2af80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:08.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 >> v1:172.21.15.19:6789/0 conn(0x7f2af80b4d60 legacy=0x7f2af813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:08.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 10 monclient: _finish_hunting 0 2024-03-25T19:33:08.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 1 monclient: found mon.noname-b 2024-03-25T19:33:08.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:08.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:33:08.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af80b1030 con 0x7f2af814be90 2024-03-25T19:33:08.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 10 monclient: _finish_auth 0 2024-03-25T19:33:08.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 10 monclient: _check_auth_tickets 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2af0001a10 con 0x7f2af814be90 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2af659c640 10 monclient: dump: 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:08.450 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:08.451 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:08.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2afd086640 5 monclient: authenticate success, global_id 4385 2024-03-25T19:33:08.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.449+0000 7f2afd086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.477+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2af0001b70 con 0x7f2af814be90 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.477+0000 7f2af659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.477+0000 7f2af659c640 1 -- 172.21.15.19:0/2868085662 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2af0001f80 con 0x7f2af814be90 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.477+0000 7f2af659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.477+0000 7f2af659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.477+0000 7f2af659c640 10 monclient: dump: 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:08.478 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:08.478 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:08.478 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:08.478 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:08.478 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:08.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.477+0000 7f2afd086640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:08.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.477+0000 7f2afd086640 4 set_mon_vals no callback set 2024-03-25T19:33:08.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.478+0000 7f2afd086640 10 monclient: shutdown 2024-03-25T19:33:08.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.478+0000 7f2afd086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:08.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.478+0000 7f2afd086640 1 -- 172.21.15.19:0/2868085662 >> v1:172.21.15.19:6790/0 conn(0x7f2af814be90 legacy=0x7f2af814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:08.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.478+0000 7f2afd086640 1 -- 172.21.15.19:0/2868085662 shutdown_connections 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.478+0000 7f2afd086640 1 -- 172.21.15.19:0/2868085662 >> 172.21.15.19:0/2868085662 conn(0x7f2af80617a0 msgr2=0x7f2af8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.478+0000 7f2afd086640 1 -- 172.21.15.19:0/2868085662 shutdown_connections 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.478+0000 7f2afd086640 1 -- 172.21.15.19:0/2868085662 wait complete. 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.479+0000 7f2afd086640 10 monclient: build_initial_monmap 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.479+0000 7f2afd086640 10 monclient: monmap: 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.479+0000 7f2afd086640 1 librados: starting msgr at 2024-03-25T19:33:08.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.479+0000 7f2afd086640 1 librados: starting objecter 2024-03-25T19:33:08.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.479+0000 7f2afd086640 1 Processor -- start 2024-03-25T19:33:08.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.479+0000 7f2afd086640 1 -- start start 2024-03-25T19:33:08.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.479+0000 7f2afd086640 1 librados: setting wanted keys 2024-03-25T19:33:08.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.479+0000 7f2afd086640 1 librados: calling monclient init 2024-03-25T19:33:08.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.479+0000 7f2afd086640 10 monclient: init 2024-03-25T19:33:08.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.480+0000 7f2afd086640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:08.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.480+0000 7f2afd086640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:08.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.480+0000 7f2afd086640 10 monclient(hunting): picked mon.noname-c con 0x7f2af80b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:08.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.480+0000 7f2afd086640 10 monclient(hunting): picked mon.noname-a con 0x7f2af80b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:08.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.480+0000 7f2afd086640 10 monclient(hunting): picked mon.noname-b con 0x7f2af814be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:08.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.480+0000 7f2afd086640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af8137220 con 0x7f2af80b4d60 2024-03-25T19:33:08.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.480+0000 7f2afd086640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af8145330 con 0x7f2af814be90 2024-03-25T19:33:08.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.480+0000 7f2afd086640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af813e750 con 0x7f2af80b1730 2024-03-25T19:33:08.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.480+0000 7f2afd086640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:08.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.480+0000 7f2afd086640 10 monclient(hunting): authenticate will time out at 6241.750583s 2024-03-25T19:33:08.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.494+0000 7f2af759e640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f2af814be90 0x7f2af8144c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:43676/0 (socket says 172.21.15.19:43676) 2024-03-25T19:33:08.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.494+0000 7f2af759e640 1 -- 172.21.15.19:0/3497778441 learned_addr learned my addr 172.21.15.19:0/3497778441 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:08.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.494+0000 7f2aef7fe640 10 client.?.objecter ms_handle_connect 0x7f2af814be90 2024-03-25T19:33:08.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.494+0000 7f2aef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:08.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.495+0000 7f2aef7fe640 10 client.?.objecter ms_handle_connect 0x7f2af80b4d60 2024-03-25T19:33:08.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.495+0000 7f2aef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:08.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.500+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743275352 0 0) 0x7f2af8145330 con 0x7f2af814be90 2024-03-25T19:33:08.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.500+0000 7f2aef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:08.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.500+0000 7f2aef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:08.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.500+0000 7f2aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:08.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.500+0000 7f2aef7fe640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:33:08.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.500+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad8004eb0 con 0x7f2af814be90 2024-03-25T19:33:08.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.500+0000 7f2aef7fe640 10 client.?.objecter ms_handle_connect 0x7f2af80b1730 2024-03-25T19:33:08.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.500+0000 7f2aef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:08.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.504+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890801664 0 0) 0x7f2af8137220 con 0x7f2af80b4d60 2024-03-25T19:33:08.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.504+0000 7f2aef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:08.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.504+0000 7f2aef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:08.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.504+0000 7f2aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:08.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.504+0000 7f2aef7fe640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:33:08.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.504+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af8145330 con 0x7f2af80b4d60 2024-03-25T19:33:08.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.509+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923206521 0 0) 0x7f2af813e750 con 0x7f2af80b1730 2024-03-25T19:33:08.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.509+0000 7f2aef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:08.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.509+0000 7f2aef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:08.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.509+0000 7f2aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:08.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.509+0000 7f2aef7fe640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:33:08.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.509+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af8137220 con 0x7f2af80b1730 2024-03-25T19:33:08.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.523+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3312928066 0 0) 0x7f2af8145330 con 0x7f2af80b4d60 2024-03-25T19:33:08.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.523+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2af813e750 con 0x7f2af80b4d60 2024-03-25T19:33:08.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.523+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2ae0001870 con 0x7f2af80b4d60 2024-03-25T19:33:08.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.523+0000 7f2aef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.523+0000 7f2aef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.523+0000 7f2aef7fe640 10 monclient(hunting): dump: 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:08.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.523+0000 7f2aef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:08.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.526+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3032285405 0 0) 0x7f2ad8004eb0 con 0x7f2af814be90 2024-03-25T19:33:08.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.526+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2af8145330 con 0x7f2af814be90 2024-03-25T19:33:08.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.526+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2af0001f40 con 0x7f2af814be90 2024-03-25T19:33:08.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.526+0000 7f2aef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:08.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.526+0000 7f2aef7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:08.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.526+0000 7f2aef7fe640 10 monclient(hunting): dump: 2024-03-25T19:33:08.526 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:08.526 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:08.526 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:08.526 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:08.526 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:08.527 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:08.527 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:08.527 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:08.527 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:08.527 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:08.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2312658670 0 0) 0x7f2af8145330 con 0x7f2af814be90 2024-03-25T19:33:08.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 >> v1:172.21.15.19:6790/0 conn(0x7f2af80b1730 legacy=0x7f2af81dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 >> v1:172.21.15.19:6789/0 conn(0x7f2af80b4d60 legacy=0x7f2af8136b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2aef7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2aef7fe640 1 monclient: found mon.b 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2aef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2aef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af81dba10 con 0x7f2af814be90 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2aef7fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2aef7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2aef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2afd086640 5 monclient: authenticate success, global_id 4408 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2afd086640 10 monclient: _renew_subs 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2afd086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2afd086640 1 -- 172.21.15.19:0/3497778441 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af813dba0 con 0x7f2af814be90 2024-03-25T19:33:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2afd086640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:08.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2afd086640 10 monclient: _renew_subs 2024-03-25T19:33:08.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2afd086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:08.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2afd086640 1 -- 172.21.15.19:0/3497778441 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af813ee50 con 0x7f2af814be90 2024-03-25T19:33:08.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.541+0000 7f2afd086640 1 librados: init done 2024-03-25T19:33:08.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.542+0000 7f2afd086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:08.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.542+0000 7f2afd086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:08.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.542+0000 7f2afd086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:08.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.542+0000 7f2afd086640 1 -- 172.21.15.19:0/3497778441 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af813f100 con 0x7f2af814be90 2024-03-25T19:33:08.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.824+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2af0001510 con 0x7f2af814be90 2024-03-25T19:33:08.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.824+0000 7f2aef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:08.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.824+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2af0003dc0 con 0x7f2af814be90 2024-03-25T19:33:08.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.824+0000 7f2aef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:08.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.824+0000 7f2aef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:08.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.824+0000 7f2aef7fe640 10 monclient: dump: 2024-03-25T19:33:08.824 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:08.824 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:08.824 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:08.824 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:08.824 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:08.824 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:08.825 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:08.825 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:08.825 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:08.825 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:08.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.824+0000 7f2af5d9b640 4 set_mon_vals no callback set 2024-03-25T19:33:08.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.824+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f2af0003fe0 con 0x7f2af814be90 2024-03-25T19:33:08.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.825+0000 7f2af6d9d640 10 monclient: get_auth_request con 0x7f2ad806dfb0 auth_method 0 2024-03-25T19:33:08.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.825+0000 7f2aef7fe640 10 client.4408.objecter ms_handle_connect 0x7f2ad806dfb0 2024-03-25T19:33:08.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.829+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f2af0054a90 con 0x7f2af814be90 2024-03-25T19:33:08.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.829+0000 7f2aef7fe640 10 client.4408.objecter ms_dispatch 0x7f2af8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:08.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.829+0000 7f2aef7fe640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:08.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.829+0000 7f2aef7fe640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:08.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.829+0000 7f2aef7fe640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-25T19:33:08.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.829+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2af00505a0 con 0x7f2af814be90 2024-03-25T19:33:08.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.829+0000 7f2aef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:08.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:08.829+0000 7f2aef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:09.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.060+0000 7f2afd086640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:09.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.060+0000 7f2afd086640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:09.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.060+0000 7f2afd086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:09.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.060+0000 7f2afd086640 1 -- 172.21.15.19:0/3497778441 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2af813e8e0 con 0x7f2af814be90 2024-03-25T19:33:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.093+0000 7f2aef7fe640 1 -- 172.21.15.19:0/3497778441 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f2af00524d0 con 0x7f2af814be90 2024-03-25T19:33:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.094+0000 7f2aef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.094+0000 7f2aef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:09.094 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:33:09.094 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:09.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 10 librados: watch_flush enter 2024-03-25T19:33:09.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 10 librados: watch_flush exit 2024-03-25T19:33:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-25T19:33:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 10 client.4408.objecter successfully canceled tick 2024-03-25T19:33:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 1 -- 172.21.15.19:0/3497778441 >> v1:172.21.15.188:6800/2207506229 conn(0x7f2ad806dfb0 legacy=0x7f2ad8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 10 monclient: shutdown 2024-03-25T19:33:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 1 -- 172.21.15.19:0/3497778441 >> v1:172.21.15.188:6789/0 conn(0x7f2af814be90 legacy=0x7f2af8144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 1 -- 172.21.15.19:0/3497778441 shutdown_connections 2024-03-25T19:33:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 1 -- 172.21.15.19:0/3497778441 >> 172.21.15.19:0/3497778441 conn(0x7f2af80617a0 msgr2=0x7f2af80b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 1 -- 172.21.15.19:0/3497778441 shutdown_connections 2024-03-25T19:33:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 1 -- 172.21.15.19:0/3497778441 wait complete. 2024-03-25T19:33:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:09.096+0000 7f2afd086640 1 librados: shutdown 2024-03-25T19:33:09.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:12.354 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:04:35.084 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.359 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:04:35.089 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.389 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:04:35.119 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.417 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:04:35.120 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.420 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:04:35.124 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.454 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:04:35.183 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.463 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:04:35.192 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.466 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:04:35.169 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.492 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:04:35.222 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:15.916 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:16.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.030+0000 7f8358ad9640 10 monclient: get_monmap_and_config 2024-03-25T19:33:16.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.030+0000 7f8358ad9640 10 monclient: build_initial_monmap 2024-03-25T19:33:16.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.030+0000 7f8358ad9640 10 monclient: monmap: 2024-03-25T19:33:16.030 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:16.030 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:16.031 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:33:16.032000+0000 2024-03-25T19:33:16.031 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:33:16.032000+0000 2024-03-25T19:33:16.031 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:16.031 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:16.031 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:16.031 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:33:16.031 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:33:16.031 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:16.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.031+0000 7f8358ad9640 1 Processor -- start 2024-03-25T19:33:16.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.031+0000 7f8358ad9640 1 -- start start 2024-03-25T19:33:16.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.031+0000 7f8358ad9640 10 monclient: init 2024-03-25T19:33:16.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.032+0000 7f8358ad9640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:16.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.032+0000 7f8358ad9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:16.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.032+0000 7f8358ad9640 10 monclient(hunting): picked mon.noname-b con 0x7f83540a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:16.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.032+0000 7f8358ad9640 10 monclient(hunting): picked mon.noname-a con 0x7f835413f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:16.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.032+0000 7f8358ad9640 10 monclient(hunting): picked mon.noname-c con 0x7f83541430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:16.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.032+0000 7f8358ad9640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8354142520 con 0x7f835413f710 2024-03-25T19:33:16.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.032+0000 7f8358ad9640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83540b28a0 con 0x7f83541430c0 2024-03-25T19:33:16.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.032+0000 7f8358ad9640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83540b2a30 con 0x7f83540a98f0 2024-03-25T19:33:16.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.032+0000 7f8358ad9640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:16.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.032+0000 7f8358ad9640 10 monclient(hunting): authenticate will time out at 6249.302442s 2024-03-25T19:33:16.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.041+0000 7f8351d9b640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f835413f710 0x7f83541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:46720/0 (socket says 172.21.15.19:46720) 2024-03-25T19:33:16.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.041+0000 7f8351d9b640 1 -- 172.21.15.19:0/2677564426 learned_addr learned my addr 172.21.15.19:0/2677564426 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:16.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.061+0000 7f835159a640 1 -- 172.21.15.19:0/2677564426 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56717251 0 0) 0x7f83540b28a0 con 0x7f83541430c0 2024-03-25T19:33:16.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.061+0000 7f835159a640 10 monclient(hunting): _negotiate 2024-03-25T19:33:16.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.061+0000 7f835159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:16.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.061+0000 7f835159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:16.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.061+0000 7f835159a640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:33:16.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.061+0000 7f835159a640 1 -- 172.21.15.19:0/2677564426 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f833c003520 con 0x7f83541430c0 2024-03-25T19:33:16.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.069+0000 7f835159a640 1 -- 172.21.15.19:0/2677564426 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818352462 0 0) 0x7f83540b2a30 con 0x7f83540a98f0 2024-03-25T19:33:16.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.069+0000 7f835159a640 10 monclient(hunting): _negotiate 2024-03-25T19:33:16.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.069+0000 7f835159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:16.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.069+0000 7f835159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:16.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.069+0000 7f835159a640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:33:16.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.069+0000 7f835159a640 1 -- 172.21.15.19:0/2677564426 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83540b28a0 con 0x7f83540a98f0 2024-03-25T19:33:16.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 1 -- 172.21.15.19:0/2677564426 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 77850872 0 0) 0x7f833c003520 con 0x7f83541430c0 2024-03-25T19:33:16.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 1 -- 172.21.15.19:0/2677564426 >> v1:172.21.15.19:6790/0 conn(0x7f83540a98f0 legacy=0x7f83540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:16.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 1 -- 172.21.15.19:0/2677564426 >> v1:172.21.15.19:6789/0 conn(0x7f835413f710 legacy=0x7f83541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:16.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 10 monclient: _finish_hunting 0 2024-03-25T19:33:16.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 1 monclient: found mon.noname-c 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 1 -- 172.21.15.19:0/2677564426 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83540b44c0 con 0x7f83541430c0 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 10 monclient: _finish_auth 0 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 10 monclient: _check_auth_tickets 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 1 -- 172.21.15.19:0/2677564426 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f834c001a10 con 0x7f83541430c0 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f835159a640 10 monclient: dump: 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:16.077 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:16.078 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:16.078 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:16.078 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:16.078 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:16.078 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:16.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f8358ad9640 5 monclient: authenticate success, global_id 4411 2024-03-25T19:33:16.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.076+0000 7f8358ad9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:16.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.096+0000 7f835159a640 1 -- 172.21.15.19:0/2677564426 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f834c001b70 con 0x7f83541430c0 2024-03-25T19:33:16.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.096+0000 7f835159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:16.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.096+0000 7f835159a640 1 -- 172.21.15.19:0/2677564426 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f834c001f80 con 0x7f83541430c0 2024-03-25T19:33:16.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.096+0000 7f835159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.096+0000 7f835159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.096+0000 7f835159a640 10 monclient: dump: 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.096+0000 7f8358ad9640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:16.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.096+0000 7f8358ad9640 4 set_mon_vals no callback set 2024-03-25T19:33:16.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.097+0000 7f8358ad9640 10 monclient: shutdown 2024-03-25T19:33:16.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.097+0000 7f8358ad9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:16.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.097+0000 7f8358ad9640 1 -- 172.21.15.19:0/2677564426 >> v1:172.21.15.188:6789/0 conn(0x7f83541430c0 legacy=0x7f835414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:16.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.097+0000 7f8358ad9640 1 -- 172.21.15.19:0/2677564426 shutdown_connections 2024-03-25T19:33:16.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.097+0000 7f8358ad9640 1 -- 172.21.15.19:0/2677564426 >> 172.21.15.19:0/2677564426 conn(0x7f83540b7ea0 msgr2=0x7f83540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:16.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.097+0000 7f8358ad9640 1 -- 172.21.15.19:0/2677564426 shutdown_connections 2024-03-25T19:33:16.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.097+0000 7f8358ad9640 1 -- 172.21.15.19:0/2677564426 wait complete. 2024-03-25T19:33:16.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.098+0000 7f8358ad9640 10 monclient: build_initial_monmap 2024-03-25T19:33:16.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.098+0000 7f8358ad9640 10 monclient: monmap: 2024-03-25T19:33:16.099 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.099+0000 7f8358ad9640 1 librados: starting msgr at 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.099+0000 7f8358ad9640 1 librados: starting objecter 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.099+0000 7f8358ad9640 1 Processor -- start 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.099+0000 7f8358ad9640 1 -- start start 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.099+0000 7f8358ad9640 1 librados: setting wanted keys 2024-03-25T19:33:16.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.099+0000 7f8358ad9640 1 librados: calling monclient init 2024-03-25T19:33:16.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.099+0000 7f8358ad9640 10 monclient: init 2024-03-25T19:33:16.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.100+0000 7f8358ad9640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:16.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.100+0000 7f8358ad9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:16.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.100+0000 7f8358ad9640 10 monclient(hunting): picked mon.noname-b con 0x7f83540a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:16.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.100+0000 7f8358ad9640 10 monclient(hunting): picked mon.noname-a con 0x7f835413f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:16.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.100+0000 7f8358ad9640 10 monclient(hunting): picked mon.noname-c con 0x7f83541430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:16.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.100+0000 7f8358ad9640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83540ab920 con 0x7f835413f710 2024-03-25T19:33:16.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.100+0000 7f8358ad9640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8354135140 con 0x7f83540a98f0 2024-03-25T19:33:16.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.100+0000 7f8358ad9640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8354134e10 con 0x7f83541430c0 2024-03-25T19:33:16.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.100+0000 7f8358ad9640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:16.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.100+0000 7f8358ad9640 10 monclient(hunting): authenticate will time out at 6249.370127s 2024-03-25T19:33:16.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.103+0000 7f8352d9d640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f83541430c0 0x7f8354135660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:33852/0 (socket says 172.21.15.19:33852) 2024-03-25T19:33:16.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.103+0000 7f8352d9d640 1 -- 172.21.15.19:0/584238548 learned_addr learned my addr 172.21.15.19:0/584238548 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:16.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.103+0000 7f8342ffd640 10 client.?.objecter ms_handle_connect 0x7f83541430c0 2024-03-25T19:33:16.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.103+0000 7f8342ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:16.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.113+0000 7f8342ffd640 10 client.?.objecter ms_handle_connect 0x7f83540a98f0 2024-03-25T19:33:16.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.113+0000 7f8342ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:16.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.120+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748729250 0 0) 0x7f8354135140 con 0x7f83540a98f0 2024-03-25T19:33:16.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.120+0000 7f8342ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:16.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.120+0000 7f8342ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:16.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.120+0000 7f8342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:16.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.120+0000 7f8342ffd640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:33:16.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.120+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8334004eb0 con 0x7f83540a98f0 2024-03-25T19:33:16.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.123+0000 7f8342ffd640 10 client.?.objecter ms_handle_connect 0x7f835413f710 2024-03-25T19:33:16.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.123+0000 7f8342ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:16.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355471812 0 0) 0x7f8354134e10 con 0x7f83541430c0 2024-03-25T19:33:16.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:16.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:16.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:16.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:33:16.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8354135140 con 0x7f83541430c0 2024-03-25T19:33:16.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3381389654 0 0) 0x7f8334004eb0 con 0x7f83540a98f0 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8354134e10 con 0x7f83540a98f0 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8344001930 con 0x7f83540a98f0 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 10 monclient(hunting): dump: 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:16.132 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:16.133 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:16.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.131+0000 7f8342ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:16.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.135+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284437830 0 0) 0x7f83540ab920 con 0x7f835413f710 2024-03-25T19:33:16.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.135+0000 7f8342ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:16.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.135+0000 7f8342ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:16.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.135+0000 7f8342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:16.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.135+0000 7f8342ffd640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:33:16.136 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.135+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8334004eb0 con 0x7f835413f710 2024-03-25T19:33:16.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.154+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3012350261 0 0) 0x7f8354135140 con 0x7f83541430c0 2024-03-25T19:33:16.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.154+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83540ab920 con 0x7f83541430c0 2024-03-25T19:33:16.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.154+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f834c001f40 con 0x7f83541430c0 2024-03-25T19:33:16.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.154+0000 7f8342ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:16.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.154+0000 7f8342ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:16.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.154+0000 7f8342ffd640 10 monclient(hunting): dump: 2024-03-25T19:33:16.154 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:16.155 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:16.155 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:16.155 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:16.155 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:16.155 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:16.155 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:16.155 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:16.155 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:16.155 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:16.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.167+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3724679964 0 0) 0x7f8334004eb0 con 0x7f835413f710 2024-03-25T19:33:16.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.167+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8354135140 con 0x7f835413f710 2024-03-25T19:33:16.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.167+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8348001860 con 0x7f835413f710 2024-03-25T19:33:16.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.167+0000 7f8342ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:16.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.167+0000 7f8342ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:16.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.167+0000 7f8342ffd640 10 monclient(hunting): dump: 2024-03-25T19:33:16.168 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:16.168 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:16.168 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:16.168 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:16.168 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:16.168 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:16.168 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:16.168 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:16.168 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:16.168 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:16.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 909977680 0 0) 0x7f8354134e10 con 0x7f83540a98f0 2024-03-25T19:33:16.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 >> v1:172.21.15.19:6790/0 conn(0x7f83541430c0 legacy=0x7f8354135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:16.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 >> v1:172.21.15.19:6789/0 conn(0x7f835413f710 legacy=0x7f8354133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:16.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8342ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:16.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8342ffd640 1 monclient: found mon.b 2024-03-25T19:33:16.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8342ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:16.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8342ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:16.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8354137840 con 0x7f83540a98f0 2024-03-25T19:33:16.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8342ffd640 10 monclient: _finish_auth 0 2024-03-25T19:33:16.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8342ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8342ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8358ad9640 5 monclient: authenticate success, global_id 4414 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8358ad9640 10 monclient: _renew_subs 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8358ad9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8358ad9640 1 -- 172.21.15.19:0/584238548 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8354137a90 con 0x7f83540a98f0 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8358ad9640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8358ad9640 10 monclient: _renew_subs 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8358ad9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8358ad9640 1 -- 172.21.15.19:0/584238548 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83541347e0 con 0x7f83540a98f0 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.168+0000 7f8358ad9640 1 librados: init done 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.169+0000 7f8358ad9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.169+0000 7f8358ad9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.169+0000 7f8358ad9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:16.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.169+0000 7f8358ad9640 1 -- 172.21.15.19:0/584238548 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8354134fa0 con 0x7f83540a98f0 2024-03-25T19:33:16.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.464+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8344003610 con 0x7f83540a98f0 2024-03-25T19:33:16.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.464+0000 7f8342ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:16.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.464+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8344003780 con 0x7f83540a98f0 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.464+0000 7f8342ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.464+0000 7f8342ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.464+0000 7f8342ffd640 10 monclient: dump: 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:16.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.464+0000 7f8350d99640 4 set_mon_vals no callback set 2024-03-25T19:33:16.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.465+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f8344019790 con 0x7f83540a98f0 2024-03-25T19:33:16.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.466+0000 7f8351d9b640 10 monclient: get_auth_request con 0x7f833406e110 auth_method 0 2024-03-25T19:33:16.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.466+0000 7f8342ffd640 10 client.4414.objecter ms_handle_connect 0x7f833406e110 2024-03-25T19:33:16.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.470+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f83440534b0 con 0x7f83540a98f0 2024-03-25T19:33:16.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.470+0000 7f8342ffd640 10 client.4414.objecter ms_dispatch 0x7f8354000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:16.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.470+0000 7f8342ffd640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:16.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.470+0000 7f8342ffd640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:16.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.470+0000 7f8342ffd640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T19:33:16.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.471+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f834404edf0 con 0x7f83540a98f0 2024-03-25T19:33:16.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.472+0000 7f8342ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:16.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.472+0000 7f8342ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:16.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.726+0000 7f8358ad9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:16.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.726+0000 7f8358ad9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:16.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.726+0000 7f8358ad9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:16.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.726+0000 7f8358ad9640 1 -- 172.21.15.19:0/584238548 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83540627b0 con 0x7f83540a98f0 2024-03-25T19:33:16.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.760+0000 7f8342ffd640 1 -- 172.21.15.19:0/584238548 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f8344050ef0 con 0x7f83540a98f0 2024-03-25T19:33:16.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.760+0000 7f8342ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:16.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.760+0000 7f8342ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:16.760 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:33:16.760 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:16.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.762+0000 7f8358ad9640 10 librados: watch_flush enter 2024-03-25T19:33:16.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.762+0000 7f8358ad9640 10 librados: watch_flush exit 2024-03-25T19:33:16.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.762+0000 7f8358ad9640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T19:33:16.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.762+0000 7f8358ad9640 10 client.4414.objecter successfully canceled tick 2024-03-25T19:33:16.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.762+0000 7f8358ad9640 1 -- 172.21.15.19:0/584238548 >> v1:172.21.15.188:6800/2207506229 conn(0x7f833406e110 legacy=0x7f83340705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:16.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.762+0000 7f8358ad9640 10 monclient: shutdown 2024-03-25T19:33:16.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.762+0000 7f8358ad9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:16.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.762+0000 7f8358ad9640 1 -- 172.21.15.19:0/584238548 >> v1:172.21.15.188:6789/0 conn(0x7f83540a98f0 legacy=0x7f83541385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:16.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.762+0000 7f8358ad9640 1 -- 172.21.15.19:0/584238548 shutdown_connections 2024-03-25T19:33:16.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.762+0000 7f8358ad9640 1 -- 172.21.15.19:0/584238548 >> 172.21.15.19:0/584238548 conn(0x7f83540b7ea0 msgr2=0x7f835413ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:16.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.762+0000 7f8358ad9640 1 -- 172.21.15.19:0/584238548 shutdown_connections 2024-03-25T19:33:16.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.763+0000 7f8358ad9640 1 -- 172.21.15.19:0/584238548 wait complete. 2024-03-25T19:33:16.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:16.763+0000 7f8358ad9640 1 librados: shutdown 2024-03-25T19:33:16.775 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:23.581 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:23.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.696+0000 7f9b7c939640 10 monclient: get_monmap_and_config 2024-03-25T19:33:23.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.696+0000 7f9b7c939640 10 monclient: build_initial_monmap 2024-03-25T19:33:23.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.696+0000 7f9b7c939640 10 monclient: monmap: 2024-03-25T19:33:23.696 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:23.696 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:23.696 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:33:23.697388+0000 2024-03-25T19:33:23.696 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:33:23.697388+0000 2024-03-25T19:33:23.696 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:23.696 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:23.697 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:23.697 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:33:23.697 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:33:23.697 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:23.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.697+0000 7f9b7c939640 1 Processor -- start 2024-03-25T19:33:23.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.697+0000 7f9b7c939640 1 -- start start 2024-03-25T19:33:23.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.697+0000 7f9b7c939640 10 monclient: init 2024-03-25T19:33:23.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.698+0000 7f9b7c939640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:23.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.698+0000 7f9b7c939640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:23.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.698+0000 7f9b7c939640 10 monclient(hunting): picked mon.noname-a con 0x7f9b780a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:23.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.698+0000 7f9b7c939640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7813f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:23.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.698+0000 7f9b7c939640 10 monclient(hunting): picked mon.noname-b con 0x7f9b781430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:23.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.698+0000 7f9b7c939640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b78142520 con 0x7f9b780a98f0 2024-03-25T19:33:23.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.698+0000 7f9b7c939640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b780b28a0 con 0x7f9b7813f710 2024-03-25T19:33:23.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.698+0000 7f9b7c939640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b780b2a30 con 0x7f9b781430c0 2024-03-25T19:33:23.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.698+0000 7f9b7c939640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:23.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.698+0000 7f9b7c939640 10 monclient(hunting): authenticate will time out at 6256.968200s 2024-03-25T19:33:23.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.709+0000 7f9b76d9d640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f9b781430c0 0x7f9b7814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:60300/0 (socket says 172.21.15.19:60300) 2024-03-25T19:33:23.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.709+0000 7f9b76d9d640 1 -- 172.21.15.19:0/957612037 learned_addr learned my addr 172.21.15.19:0/957612037 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:23.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.728+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356773846 0 0) 0x7f9b780b2a30 con 0x7f9b781430c0 2024-03-25T19:33:23.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.728+0000 7f9b75d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:23.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.728+0000 7f9b75d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:23.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.728+0000 7f9b75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:23.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.728+0000 7f9b75d9b640 10 monclient(hunting): my global_id is 4397 2024-03-25T19:33:23.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.728+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b64003520 con 0x7f9b781430c0 2024-03-25T19:33:23.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.728+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091455773 0 0) 0x7f9b78142520 con 0x7f9b780a98f0 2024-03-25T19:33:23.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.728+0000 7f9b75d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:23.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.728+0000 7f9b75d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:23.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.728+0000 7f9b75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:23.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.728+0000 7f9b75d9b640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:33:23.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.728+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b780b2a30 con 0x7f9b780a98f0 2024-03-25T19:33:23.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.736+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556947908 0 0) 0x7f9b780b28a0 con 0x7f9b7813f710 2024-03-25T19:33:23.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.736+0000 7f9b75d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:23.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.736+0000 7f9b75d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:23.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.736+0000 7f9b75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:23.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.736+0000 7f9b75d9b640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:33:23.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.736+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b78142520 con 0x7f9b7813f710 2024-03-25T19:33:23.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3708575306 0 0) 0x7f9b780b2a30 con 0x7f9b780a98f0 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 >> v1:172.21.15.19:6790/0 conn(0x7f9b781430c0 legacy=0x7f9b7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 >> v1:172.21.15.188:6789/0 conn(0x7f9b7813f710 legacy=0x7f9b781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 1 monclient: found mon.noname-a 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b780b44c0 con 0x7f9b780a98f0 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 10 monclient: _finish_auth 0 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9b68001670 con 0x7f9b780a98f0 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b7c939640 5 monclient: authenticate success, global_id 4425 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b7c939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:23.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b75d9b640 10 monclient: dump: 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:23.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.742+0000 7f9b7c939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:23.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.753+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b680017d0 con 0x7f9b780a98f0 2024-03-25T19:33:23.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.753+0000 7f9b75d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:23.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.753+0000 7f9b75d9b640 1 -- 172.21.15.19:0/957612037 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9b68001be0 con 0x7f9b780a98f0 2024-03-25T19:33:23.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.753+0000 7f9b75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:23.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.753+0000 7f9b75d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.753+0000 7f9b75d9b640 10 monclient: dump: 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.753+0000 7f9b7c939640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:23.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.753+0000 7f9b7c939640 4 set_mon_vals no callback set 2024-03-25T19:33:23.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.754+0000 7f9b7c939640 10 monclient: shutdown 2024-03-25T19:33:23.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.754+0000 7f9b7c939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:23.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.754+0000 7f9b7c939640 1 -- 172.21.15.19:0/957612037 >> v1:172.21.15.19:6789/0 conn(0x7f9b780a98f0 legacy=0x7f9b780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.754+0000 7f9b7c939640 1 -- 172.21.15.19:0/957612037 shutdown_connections 2024-03-25T19:33:23.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.754+0000 7f9b7c939640 1 -- 172.21.15.19:0/957612037 >> 172.21.15.19:0/957612037 conn(0x7f9b780b7ea0 msgr2=0x7f9b780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:23.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.754+0000 7f9b7c939640 1 -- 172.21.15.19:0/957612037 shutdown_connections 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.754+0000 7f9b7c939640 1 -- 172.21.15.19:0/957612037 wait complete. 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.755+0000 7f9b7c939640 10 monclient: build_initial_monmap 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.755+0000 7f9b7c939640 10 monclient: monmap: 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.755+0000 7f9b7c939640 1 librados: starting msgr at 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.755+0000 7f9b7c939640 1 librados: starting objecter 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.755+0000 7f9b7c939640 1 Processor -- start 2024-03-25T19:33:23.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.755+0000 7f9b7c939640 1 -- start start 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.755+0000 7f9b7c939640 1 librados: setting wanted keys 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.755+0000 7f9b7c939640 1 librados: calling monclient init 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.755+0000 7f9b7c939640 10 monclient: init 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.756+0000 7f9b7c939640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.756+0000 7f9b7c939640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.756+0000 7f9b7c939640 10 monclient(hunting): picked mon.noname-b con 0x7f9b780a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.756+0000 7f9b7c939640 10 monclient(hunting): picked mon.noname-a con 0x7f9b7813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.756+0000 7f9b7c939640 10 monclient(hunting): picked mon.noname-c con 0x7f9b781430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.756+0000 7f9b7c939640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b78138f10 con 0x7f9b7813f710 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.756+0000 7f9b7c939640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b78139970 con 0x7f9b780a98f0 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.756+0000 7f9b7c939640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b780abb50 con 0x7f9b781430c0 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.756+0000 7f9b7c939640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:23.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.756+0000 7f9b7c939640 10 monclient(hunting): authenticate will time out at 6257.026479s 2024-03-25T19:33:23.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.763+0000 7f9b6ffff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f9b7813f710 0x7f9b780ac250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:39010/0 (socket says 172.21.15.19:39010) 2024-03-25T19:33:23.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.763+0000 7f9b6ffff640 1 -- 172.21.15.19:0/2146671392 learned_addr learned my addr 172.21.15.19:0/2146671392 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:23.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.766+0000 7f9b6effd640 10 client.?.objecter ms_handle_connect 0x7f9b7813f710 2024-03-25T19:33:23.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.766+0000 7f9b6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:23.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.770+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316135275 0 0) 0x7f9b78138f10 con 0x7f9b7813f710 2024-03-25T19:33:23.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.770+0000 7f9b6effd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:23.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.770+0000 7f9b6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:23.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.770+0000 7f9b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:23.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.770+0000 7f9b6effd640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:33:23.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.770+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b58004eb0 con 0x7f9b7813f710 2024-03-25T19:33:23.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.770+0000 7f9b6effd640 10 client.?.objecter ms_handle_connect 0x7f9b781430c0 2024-03-25T19:33:23.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.770+0000 7f9b6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:23.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.774+0000 7f9b6effd640 10 client.?.objecter ms_handle_connect 0x7f9b780a98f0 2024-03-25T19:33:23.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.774+0000 7f9b6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:23.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.785+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568686574 0 0) 0x7f9b780abb50 con 0x7f9b781430c0 2024-03-25T19:33:23.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.785+0000 7f9b6effd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:23.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.785+0000 7f9b6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:23.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.785+0000 7f9b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:23.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.785+0000 7f9b6effd640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:33:23.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.785+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b78138f10 con 0x7f9b781430c0 2024-03-25T19:33:23.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.792+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752361701 0 0) 0x7f9b78139970 con 0x7f9b780a98f0 2024-03-25T19:33:23.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.792+0000 7f9b6effd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:23.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.792+0000 7f9b6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:23.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.792+0000 7f9b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:23.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.792+0000 7f9b6effd640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:33:23.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.792+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b780abb50 con 0x7f9b780a98f0 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.795+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3117837584 0 0) 0x7f9b58004eb0 con 0x7f9b7813f710 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.795+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b78139970 con 0x7f9b7813f710 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.795+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9b60001a10 con 0x7f9b7813f710 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.795+0000 7f9b6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.795+0000 7f9b6effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.795+0000 7f9b6effd640 10 monclient(hunting): dump: 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:23.795 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:23.796 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:23.796 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:23.796 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:23.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.795+0000 7f9b6effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.798+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 167780022 0 0) 0x7f9b78138f10 con 0x7f9b781430c0 2024-03-25T19:33:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.798+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b58004eb0 con 0x7f9b781430c0 2024-03-25T19:33:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.798+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9b70001a70 con 0x7f9b781430c0 2024-03-25T19:33:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.798+0000 7f9b6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:23.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.798+0000 7f9b6effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:23.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.798+0000 7f9b6effd640 10 monclient(hunting): dump: 2024-03-25T19:33:23.799 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:23.799 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:23.799 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:23.799 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:23.799 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:23.799 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:23.799 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:23.799 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:23.799 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:23.799 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.802+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3071337292 0 0) 0x7f9b780abb50 con 0x7f9b780a98f0 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.802+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b78138f10 con 0x7f9b780a98f0 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.802+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9b68001ae0 con 0x7f9b780a98f0 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.802+0000 7f9b6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.802+0000 7f9b6effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.802+0000 7f9b6effd640 10 monclient(hunting): dump: 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:23.803 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:23.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1399519154 0 0) 0x7f9b78139970 con 0x7f9b7813f710 2024-03-25T19:33:23.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 >> v1:172.21.15.19:6790/0 conn(0x7f9b781430c0 legacy=0x7f9b78138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 >> v1:172.21.15.188:6789/0 conn(0x7f9b780a98f0 legacy=0x7f9b78136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b6effd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:23.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b6effd640 1 monclient: found mon.a 2024-03-25T19:33:23.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:23.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b6effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:23.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b780abe30 con 0x7f9b7813f710 2024-03-25T19:33:23.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b6effd640 10 monclient: _finish_auth 0 2024-03-25T19:33:23.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b6effd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:23.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:23.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b7c939640 5 monclient: authenticate success, global_id 4428 2024-03-25T19:33:23.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b7c939640 10 monclient: _renew_subs 2024-03-25T19:33:23.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b7c939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:23.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b7c939640 1 -- 172.21.15.19:0/2146671392 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b78137b40 con 0x7f9b7813f710 2024-03-25T19:33:23.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b7c939640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:23.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b7c939640 10 monclient: _renew_subs 2024-03-25T19:33:23.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b7c939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:23.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b7c939640 1 -- 172.21.15.19:0/2146671392 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b78138c10 con 0x7f9b7813f710 2024-03-25T19:33:23.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.806+0000 7f9b7c939640 1 librados: init done 2024-03-25T19:33:23.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.807+0000 7f9b7c939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:23.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.807+0000 7f9b7c939640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:23.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.807+0000 7f9b7c939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:23.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:23.807+0000 7f9b7c939640 1 -- 172.21.15.19:0/2146671392 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b780627b0 con 0x7f9b7813f710 2024-03-25T19:33:24.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.025+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b60001c70 con 0x7f9b7813f710 2024-03-25T19:33:24.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.025+0000 7f9b6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:24.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.025+0000 7f9b7559a640 4 set_mon_vals no callback set 2024-03-25T19:33:24.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.121+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9b60003a60 con 0x7f9b7813f710 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.122+0000 7f9b6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.122+0000 7f9b6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.122+0000 7f9b6effd640 10 monclient: dump: 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:24.122 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:24.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.123+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f9b60003c40 con 0x7f9b7813f710 2024-03-25T19:33:24.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.123+0000 7f9b7659c640 10 monclient: get_auth_request con 0x7f9b5806e0f0 auth_method 0 2024-03-25T19:33:24.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.124+0000 7f9b6effd640 10 client.4428.objecter ms_handle_connect 0x7f9b5806e0f0 2024-03-25T19:33:24.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.131+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f9b600548e0 con 0x7f9b7813f710 2024-03-25T19:33:24.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.131+0000 7f9b6effd640 10 client.4428.objecter ms_dispatch 0x7f9b78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:24.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.131+0000 7f9b6effd640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:24.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.131+0000 7f9b6effd640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:24.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.131+0000 7f9b6effd640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-25T19:33:24.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.132+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b60050220 con 0x7f9b7813f710 2024-03-25T19:33:24.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.132+0000 7f9b6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:24.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.132+0000 7f9b6effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:24.349 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.349+0000 7f9b7c939640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:24.349 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.349+0000 7f9b7c939640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:24.349 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.349+0000 7f9b7c939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:24.349 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.349+0000 7f9b7c939640 1 -- 172.21.15.19:0/2146671392 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b7813db40 con 0x7f9b7813f710 2024-03-25T19:33:24.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.368+0000 7f9b6effd640 1 -- 172.21.15.19:0/2146671392 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f9b60052320 con 0x7f9b7813f710 2024-03-25T19:33:24.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.368+0000 7f9b6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:24.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.368+0000 7f9b6effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:24.369 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:33:24.369 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:24.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.370+0000 7f9b7c939640 10 librados: watch_flush enter 2024-03-25T19:33:24.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.370+0000 7f9b7c939640 10 librados: watch_flush exit 2024-03-25T19:33:24.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.370+0000 7f9b7c939640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-25T19:33:24.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.370+0000 7f9b7c939640 10 client.4428.objecter successfully canceled tick 2024-03-25T19:33:24.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.370+0000 7f9b7c939640 1 -- 172.21.15.19:0/2146671392 >> v1:172.21.15.188:6800/2207506229 conn(0x7f9b5806e0f0 legacy=0x7f9b580705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:24.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.371+0000 7f9b7c939640 10 monclient: shutdown 2024-03-25T19:33:24.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.371+0000 7f9b7c939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:24.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.371+0000 7f9b7c939640 1 -- 172.21.15.19:0/2146671392 >> v1:172.21.15.19:6789/0 conn(0x7f9b7813f710 legacy=0x7f9b780ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:24.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.371+0000 7f9b7c939640 1 -- 172.21.15.19:0/2146671392 shutdown_connections 2024-03-25T19:33:24.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.371+0000 7f9b7c939640 1 -- 172.21.15.19:0/2146671392 >> 172.21.15.19:0/2146671392 conn(0x7f9b780b7ea0 msgr2=0x7f9b78074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:24.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.371+0000 7f9b7c939640 1 -- 172.21.15.19:0/2146671392 shutdown_connections 2024-03-25T19:33:24.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.371+0000 7f9b7c939640 1 -- 172.21.15.19:0/2146671392 wait complete. 2024-03-25T19:33:24.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:24.371+0000 7f9b7c939640 1 librados: shutdown 2024-03-25T19:33:24.382 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:31.189 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:31.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.305+0000 7fabe70e4640 10 monclient: get_monmap_and_config 2024-03-25T19:33:31.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.305+0000 7fabe70e4640 10 monclient: build_initial_monmap 2024-03-25T19:33:31.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.305+0000 7fabe70e4640 10 monclient: monmap: 2024-03-25T19:33:31.304 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:31.304 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:31.304 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:33:31.305574+0000 2024-03-25T19:33:31.304 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:33:31.305574+0000 2024-03-25T19:33:31.304 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:31.305 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:31.305 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:31.305 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:33:31.305 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:33:31.305 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:31.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 1 Processor -- start 2024-03-25T19:33:31.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 1 -- start start 2024-03-25T19:33:31.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 10 monclient: init 2024-03-25T19:33:31.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:31.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:31.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 10 monclient(hunting): picked mon.noname-b con 0x7fabe00b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:31.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 10 monclient(hunting): picked mon.noname-c con 0x7fabe00b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:31.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 10 monclient(hunting): picked mon.noname-a con 0x7fabe014be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:31.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe00af940 con 0x7fabe014be90 2024-03-25T19:33:31.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe00b0a70 con 0x7fabe00b1730 2024-03-25T19:33:31.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe00b0d50 con 0x7fabe00b4d60 2024-03-25T19:33:31.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:31.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.306+0000 7fabe70e4640 10 monclient(hunting): authenticate will time out at 6264.575775s 2024-03-25T19:33:31.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.315+0000 7fabdffff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fabe00b1730 0x7fabe00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:43790/0 (socket says 172.21.15.19:43790) 2024-03-25T19:33:31.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.315+0000 7fabdffff640 1 -- 172.21.15.19:0/1152723553 learned_addr learned my addr 172.21.15.19:0/1152723553 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:31.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.331+0000 7fabdf7fe640 1 -- 172.21.15.19:0/1152723553 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190516172 0 0) 0x7fabe00b0a70 con 0x7fabe00b1730 2024-03-25T19:33:31.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.331+0000 7fabdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:31.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.331+0000 7fabdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:31.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.331+0000 7fabdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:31.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.331+0000 7fabdf7fe640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:33:31.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.331+0000 7fabdf7fe640 1 -- 172.21.15.19:0/1152723553 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabcc003520 con 0x7fabe00b1730 2024-03-25T19:33:31.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 1 -- 172.21.15.19:0/1152723553 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1562184424 0 0) 0x7fabcc003520 con 0x7fabe00b1730 2024-03-25T19:33:31.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 1 -- 172.21.15.19:0/1152723553 >> v1:172.21.15.19:6790/0 conn(0x7fabe00b4d60 legacy=0x7fabe013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 1 -- 172.21.15.19:0/1152723553 >> v1:172.21.15.19:6789/0 conn(0x7fabe014be90 legacy=0x7fabe014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 1 monclient: found mon.noname-c 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 1 -- 172.21.15.19:0/1152723553 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe00b1030 con 0x7fabe00b1730 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 1 -- 172.21.15.19:0/1152723553 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fabd0001690 con 0x7fabe00b1730 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabdf7fe640 10 monclient: dump: 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:31.340 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:31.341 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:31.341 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:31.341 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:31.341 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:31.341 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:31.341 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:31.341 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:31.341 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:31.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabe70e4640 5 monclient: authenticate success, global_id 4423 2024-03-25T19:33:31.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.340+0000 7fabe70e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:31.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.362+0000 7fabdf7fe640 1 -- 172.21.15.19:0/1152723553 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabd00017f0 con 0x7fabe00b1730 2024-03-25T19:33:31.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.362+0000 7fabdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:31.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.362+0000 7fabdf7fe640 1 -- 172.21.15.19:0/1152723553 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fabd0001c40 con 0x7fabe00b1730 2024-03-25T19:33:31.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.362+0000 7fabdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:31.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.362+0000 7fabdf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:31.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.362+0000 7fabdf7fe640 10 monclient: dump: 2024-03-25T19:33:31.361 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:31.361 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:31.362 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:31.362 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:31.362 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:31.362 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:31.362 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:31.362 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:31.362 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:31.362 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:31.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.362+0000 7fabe70e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:31.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.362+0000 7fabe70e4640 4 set_mon_vals no callback set 2024-03-25T19:33:31.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.363+0000 7fabe70e4640 10 monclient: shutdown 2024-03-25T19:33:31.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.363+0000 7fabe70e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:31.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.363+0000 7fabe70e4640 1 -- 172.21.15.19:0/1152723553 >> v1:172.21.15.188:6789/0 conn(0x7fabe00b1730 legacy=0x7fabe00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:31.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.363+0000 7fabe70e4640 1 -- 172.21.15.19:0/1152723553 shutdown_connections 2024-03-25T19:33:31.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.363+0000 7fabe70e4640 1 -- 172.21.15.19:0/1152723553 >> 172.21.15.19:0/1152723553 conn(0x7fabe00617a0 msgr2=0x7fabe0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:31.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.363+0000 7fabe70e4640 1 -- 172.21.15.19:0/1152723553 shutdown_connections 2024-03-25T19:33:31.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.363+0000 7fabe70e4640 1 -- 172.21.15.19:0/1152723553 wait complete. 2024-03-25T19:33:31.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.364+0000 7fabe70e4640 10 monclient: build_initial_monmap 2024-03-25T19:33:31.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.364+0000 7fabe70e4640 10 monclient: monmap: 2024-03-25T19:33:31.363 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:31.364 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:31.364 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:33:31.364 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:33:31.364 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:31.364 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:33:31.364 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:31.364 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:33:31.364 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:33:31.364 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:31.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.364+0000 7fabe70e4640 1 librados: starting msgr at 2024-03-25T19:33:31.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.364+0000 7fabe70e4640 1 librados: starting objecter 2024-03-25T19:33:31.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.365+0000 7fabe70e4640 1 Processor -- start 2024-03-25T19:33:31.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.365+0000 7fabe70e4640 1 -- start start 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.365+0000 7fabe70e4640 1 librados: setting wanted keys 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.365+0000 7fabe70e4640 1 librados: calling monclient init 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.365+0000 7fabe70e4640 10 monclient: init 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.365+0000 7fabe70e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.365+0000 7fabe70e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.365+0000 7fabe70e4640 10 monclient(hunting): picked mon.noname-c con 0x7fabe00b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.365+0000 7fabe70e4640 10 monclient(hunting): picked mon.noname-b con 0x7fabe00b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.366+0000 7fabe70e4640 10 monclient(hunting): picked mon.noname-a con 0x7fabe014be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.366+0000 7fabe70e4640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe01d83d0 con 0x7fabe014be90 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.366+0000 7fabe70e4640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe01d77a0 con 0x7fabe00b4d60 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.366+0000 7fabe70e4640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe0138270 con 0x7fabe00b1730 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.366+0000 7fabe70e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:31.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.366+0000 7fabe70e4640 10 monclient(hunting): authenticate will time out at 6264.635067s 2024-03-25T19:33:31.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.375+0000 7fabdffff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fabe00b4d60 0x7fabe01405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:43844/0 (socket says 172.21.15.19:43844) 2024-03-25T19:33:31.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.375+0000 7fabdffff640 1 -- 172.21.15.19:0/2480682530 learned_addr learned my addr 172.21.15.19:0/2480682530 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:31.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.376+0000 7fabdd7fa640 10 client.?.objecter ms_handle_connect 0x7fabe00b4d60 2024-03-25T19:33:31.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.376+0000 7fabdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:31.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.377+0000 7fabdd7fa640 10 client.?.objecter ms_handle_connect 0x7fabe00b1730 2024-03-25T19:33:31.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.377+0000 7fabdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:31.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.388+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461639729 0 0) 0x7fabe01d77a0 con 0x7fabe00b4d60 2024-03-25T19:33:31.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.388+0000 7fabdd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:31.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.388+0000 7fabdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:31.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.388+0000 7fabdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:31.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.388+0000 7fabdd7fa640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:33:31.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.388+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc0004eb0 con 0x7fabe00b4d60 2024-03-25T19:33:31.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.393+0000 7fabdd7fa640 10 client.?.objecter ms_handle_connect 0x7fabe014be90 2024-03-25T19:33:31.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.393+0000 7fabdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:31.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.398+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002510016 0 0) 0x7fabe0138270 con 0x7fabe00b1730 2024-03-25T19:33:31.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.398+0000 7fabdd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:31.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.398+0000 7fabdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:31.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.398+0000 7fabdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:31.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.398+0000 7fabdd7fa640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:33:31.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.398+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe01d77a0 con 0x7fabe00b1730 2024-03-25T19:33:31.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.401+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 60914607 0 0) 0x7fabc0004eb0 con 0x7fabe00b4d60 2024-03-25T19:33:31.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.401+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabe0138270 con 0x7fabe00b4d60 2024-03-25T19:33:31.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.401+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fabd0001bc0 con 0x7fabe00b4d60 2024-03-25T19:33:31.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.401+0000 7fabdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.401+0000 7fabdd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.401+0000 7fabdd7fa640 10 monclient(hunting): dump: 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:31.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.401+0000 7fabdd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:31.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.409+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1876984183 0 0) 0x7fabe01d77a0 con 0x7fabe00b1730 2024-03-25T19:33:31.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.409+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabc0004eb0 con 0x7fabe00b1730 2024-03-25T19:33:31.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.409+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fabc80018a0 con 0x7fabe00b1730 2024-03-25T19:33:31.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.409+0000 7fabdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:31.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.409+0000 7fabdd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:31.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.409+0000 7fabdd7fa640 10 monclient(hunting): dump: 2024-03-25T19:33:31.408 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:31.408 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:31.408 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:31.409 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:31.409 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:31.409 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:31.409 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:31.409 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:31.409 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:31.409 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:31.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3203124647 0 0) 0x7fabe0138270 con 0x7fabe00b4d60 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 >> v1:172.21.15.19:6790/0 conn(0x7fabe00b1730 legacy=0x7fabe01d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 >> v1:172.21.15.19:6789/0 conn(0x7fabe014be90 legacy=0x7fabe0142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabdd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabdd7fa640 1 monclient: found mon.b 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabdd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabdd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe0140d00 con 0x7fabe00b4d60 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabdd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabdd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabdd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabe70e4640 5 monclient: authenticate success, global_id 4426 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabe70e4640 10 monclient: _renew_subs 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabe70e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabe70e4640 1 -- 172.21.15.19:0/2480682530 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe0138a80 con 0x7fabe00b4d60 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabe70e4640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:31.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabe70e4640 10 monclient: _renew_subs 2024-03-25T19:33:31.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabe70e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:31.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabe70e4640 1 -- 172.21.15.19:0/2480682530 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe0143640 con 0x7fabe00b4d60 2024-03-25T19:33:31.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.417+0000 7fabe70e4640 1 librados: init done 2024-03-25T19:33:31.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.418+0000 7fabe70e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:31.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.418+0000 7fabe70e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:31.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.418+0000 7fabe70e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:31.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.418+0000 7fabe70e4640 1 -- 172.21.15.19:0/2480682530 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe0140ef0 con 0x7fabe00b4d60 2024-03-25T19:33:31.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.705+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabd0004bf0 con 0x7fabe00b4d60 2024-03-25T19:33:31.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.705+0000 7fabdd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.705+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fabd0004d60 con 0x7fabe00b4d60 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.705+0000 7fabdd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.705+0000 7fabdd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.705+0000 7fabdd7fa640 10 monclient: dump: 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:31.705 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:31.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.705+0000 7fabdeffd640 4 set_mon_vals no callback set 2024-03-25T19:33:31.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.706+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fabd001ad30 con 0x7fabe00b4d60 2024-03-25T19:33:31.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.707+0000 7fabe4e80640 10 monclient: get_auth_request con 0x7fabc006de40 auth_method 0 2024-03-25T19:33:31.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.707+0000 7fabdd7fa640 10 client.4426.objecter ms_handle_connect 0x7fabc006de40 2024-03-25T19:33:31.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.711+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fabd00547b0 con 0x7fabe00b4d60 2024-03-25T19:33:31.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.711+0000 7fabdd7fa640 10 client.4426.objecter ms_dispatch 0x7fabe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:31.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.711+0000 7fabdd7fa640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:31.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.711+0000 7fabdd7fa640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:31.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.711+0000 7fabdd7fa640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-25T19:33:31.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.712+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabd00502c0 con 0x7fabe00b4d60 2024-03-25T19:33:31.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.712+0000 7fabdd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:31.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.712+0000 7fabdd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:31.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.967+0000 7fabe70e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:31.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.967+0000 7fabe70e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:31.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.967+0000 7fabe70e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:31.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:31.968+0000 7fabe70e4640 1 -- 172.21.15.19:0/2480682530 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabe00afbd0 con 0x7fabe00b4d60 2024-03-25T19:33:31.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.000+0000 7fabdd7fa640 1 -- 172.21.15.19:0/2480682530 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fabd00521f0 con 0x7fabe00b4d60 2024-03-25T19:33:31.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.000+0000 7fabdd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:31.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.000+0000 7fabdd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:31.999 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:33:31.999 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:32.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.002+0000 7fabe70e4640 10 librados: watch_flush enter 2024-03-25T19:33:32.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.002+0000 7fabe70e4640 10 librados: watch_flush exit 2024-03-25T19:33:32.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.002+0000 7fabe70e4640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-25T19:33:32.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.002+0000 7fabe70e4640 10 client.4426.objecter successfully canceled tick 2024-03-25T19:33:32.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.003+0000 7fabe70e4640 1 -- 172.21.15.19:0/2480682530 >> v1:172.21.15.188:6800/2207506229 conn(0x7fabc006de40 legacy=0x7fabc0070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:32.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.003+0000 7fabe70e4640 10 monclient: shutdown 2024-03-25T19:33:32.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.003+0000 7fabe70e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:32.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.003+0000 7fabe70e4640 1 -- 172.21.15.19:0/2480682530 >> v1:172.21.15.188:6789/0 conn(0x7fabe00b4d60 legacy=0x7fabe01405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:32.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.003+0000 7fabe70e4640 1 -- 172.21.15.19:0/2480682530 shutdown_connections 2024-03-25T19:33:32.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.003+0000 7fabe70e4640 1 -- 172.21.15.19:0/2480682530 >> 172.21.15.19:0/2480682530 conn(0x7fabe00617a0 msgr2=0x7fabe0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:32.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.003+0000 7fabe70e4640 1 -- 172.21.15.19:0/2480682530 shutdown_connections 2024-03-25T19:33:32.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.003+0000 7fabe70e4640 1 -- 172.21.15.19:0/2480682530 wait complete. 2024-03-25T19:33:32.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:32.003+0000 7fabe70e4640 1 librados: shutdown 2024-03-25T19:33:32.014 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:32.387 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:04:55.116 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.442 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:04:55.171 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.474 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:04:55.203 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.505 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:04:55.208 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.516 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:04:55.245 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.517 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:04:55.219 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.520 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:04:55.250 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.547 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:04:55.250 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.560 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:04:55.289 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:38.820 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:38.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.938+0000 7f3fafb1c640 10 monclient: get_monmap_and_config 2024-03-25T19:33:38.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.938+0000 7f3fafb1c640 10 monclient: build_initial_monmap 2024-03-25T19:33:38.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.938+0000 7f3fafb1c640 10 monclient: monmap: 2024-03-25T19:33:38.937 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:38.937 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:38.938 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:33:38.938824+0000 2024-03-25T19:33:38.938 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:33:38.938824+0000 2024-03-25T19:33:38.938 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:38.938 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:38.938 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:38.938 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:33:38.938 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:33:38.938 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:38.939 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.939+0000 7f3fafb1c640 1 Processor -- start 2024-03-25T19:33:38.939 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.939+0000 7f3fafb1c640 1 -- start start 2024-03-25T19:33:38.939 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.939+0000 7f3fafb1c640 10 monclient: init 2024-03-25T19:33:38.939 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.940+0000 7f3fafb1c640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:38.939 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.940+0000 7f3fafb1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:38.939 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.940+0000 7f3fafb1c640 10 monclient(hunting): picked mon.noname-c con 0x7f3fa80a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:38.939 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.940+0000 7f3fafb1c640 10 monclient(hunting): picked mon.noname-a con 0x7f3fa813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:38.939 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.940+0000 7f3fafb1c640 10 monclient(hunting): picked mon.noname-b con 0x7f3fa81430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:38.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.940+0000 7f3fafb1c640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa8142520 con 0x7f3fa813f710 2024-03-25T19:33:38.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.940+0000 7f3fafb1c640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa80b28a0 con 0x7f3fa80a98f0 2024-03-25T19:33:38.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.940+0000 7f3fafb1c640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa80b2a30 con 0x7f3fa81430c0 2024-03-25T19:33:38.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.940+0000 7f3fafb1c640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:38.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.940+0000 7f3fafb1c640 10 monclient(hunting): authenticate will time out at 6272.209136s 2024-03-25T19:33:38.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.949+0000 7f3fad8b8640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f3fa80a98f0 0x7f3fa80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:48630/0 (socket says 172.21.15.19:48630) 2024-03-25T19:33:38.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.950+0000 7f3fad8b8640 1 -- 172.21.15.19:0/593513153 learned_addr learned my addr 172.21.15.19:0/593513153 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:38.965 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.966+0000 7f3fac8b6640 1 -- 172.21.15.19:0/593513153 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314512191 0 0) 0x7f3fa8142520 con 0x7f3fa813f710 2024-03-25T19:33:38.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.966+0000 7f3fac8b6640 10 monclient(hunting): _negotiate 2024-03-25T19:33:38.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.966+0000 7f3fac8b6640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:38.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.966+0000 7f3fac8b6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:38.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.966+0000 7f3fac8b6640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:33:38.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:38.966+0000 7f3fac8b6640 1 -- 172.21.15.19:0/593513153 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f94003520 con 0x7f3fa813f710 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 1 -- 172.21.15.19:0/593513153 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2036154314 0 0) 0x7f3f94003520 con 0x7f3fa813f710 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 1 -- 172.21.15.19:0/593513153 >> v1:172.21.15.19:6790/0 conn(0x7f3fa81430c0 legacy=0x7f3fa814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 1 -- 172.21.15.19:0/593513153 >> v1:172.21.15.188:6789/0 conn(0x7f3fa80a98f0 legacy=0x7f3fa80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 10 monclient: _finish_hunting 0 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 1 monclient: found mon.noname-a 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 1 -- 172.21.15.19:0/593513153 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa80b44c0 con 0x7f3fa813f710 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 10 monclient: _finish_auth 0 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 10 monclient: _check_auth_tickets 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 1 -- 172.21.15.19:0/593513153 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3f98001690 con 0x7f3fa813f710 2024-03-25T19:33:39.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.064+0000 7f3fac8b6640 10 monclient: dump: 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.065+0000 7f3fafb1c640 5 monclient: authenticate success, global_id 4437 2024-03-25T19:33:39.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.065+0000 7f3fafb1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fac8b6640 1 -- 172.21.15.19:0/593513153 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f980017f0 con 0x7f3fa813f710 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fac8b6640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fac8b6640 1 -- 172.21.15.19:0/593513153 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3f98001c00 con 0x7f3fa813f710 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fac8b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fac8b6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fac8b6640 10 monclient: dump: 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:39.090 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:39.091 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:39.091 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:39.091 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:39.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fafb1c640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:39.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fafb1c640 4 set_mon_vals no callback set 2024-03-25T19:33:39.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fafb1c640 10 monclient: shutdown 2024-03-25T19:33:39.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fafb1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:39.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fafb1c640 1 -- 172.21.15.19:0/593513153 >> v1:172.21.15.19:6789/0 conn(0x7f3fa813f710 legacy=0x7f3fa81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:39.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fafb1c640 1 -- 172.21.15.19:0/593513153 shutdown_connections 2024-03-25T19:33:39.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fafb1c640 1 -- 172.21.15.19:0/593513153 >> 172.21.15.19:0/593513153 conn(0x7f3fa80b7ea0 msgr2=0x7f3fa80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:39.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.091+0000 7f3fafb1c640 1 -- 172.21.15.19:0/593513153 shutdown_connections 2024-03-25T19:33:39.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.092+0000 7f3fafb1c640 1 -- 172.21.15.19:0/593513153 wait complete. 2024-03-25T19:33:39.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.092+0000 7f3fafb1c640 10 monclient: build_initial_monmap 2024-03-25T19:33:39.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.092+0000 7f3fafb1c640 10 monclient: monmap: 2024-03-25T19:33:39.093 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:39.093 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:39.093 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.092+0000 7f3fafb1c640 1 librados: starting msgr at 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.092+0000 7f3fafb1c640 1 librados: starting objecter 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 1 Processor -- start 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 1 -- start start 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 1 librados: setting wanted keys 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 1 librados: calling monclient init 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 10 monclient: init 2024-03-25T19:33:39.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:39.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:39.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 10 monclient(hunting): picked mon.noname-b con 0x7f3fa80a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:39.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 10 monclient(hunting): picked mon.noname-a con 0x7f3fa813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:39.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 10 monclient(hunting): picked mon.noname-c con 0x7f3fa81430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:39.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa80abac0 con 0x7f3fa813f710 2024-03-25T19:33:39.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa80abda0 con 0x7f3fa80a98f0 2024-03-25T19:33:39.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa81380e0 con 0x7f3fa81430c0 2024-03-25T19:33:39.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:39.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.093+0000 7f3fafb1c640 10 monclient(hunting): authenticate will time out at 6272.362555s 2024-03-25T19:33:39.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.109+0000 7f3fad0b7640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f3fa813f710 0x7f3fa8138b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:37910/0 (socket says 172.21.15.19:37910) 2024-03-25T19:33:39.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.109+0000 7f3fad0b7640 1 -- 172.21.15.19:0/2003041943 learned_addr learned my addr 172.21.15.19:0/2003041943 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:39.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.110+0000 7f3f9e7fc640 10 client.?.objecter ms_handle_connect 0x7f3fa813f710 2024-03-25T19:33:39.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.110+0000 7f3f9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:39.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.114+0000 7f3f9e7fc640 10 client.?.objecter ms_handle_connect 0x7f3fa80a98f0 2024-03-25T19:33:39.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.114+0000 7f3f9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:39.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.124+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396255141 0 0) 0x7f3fa80abac0 con 0x7f3fa813f710 2024-03-25T19:33:39.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.124+0000 7f3f9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:39.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.124+0000 7f3f9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:39.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.124+0000 7f3f9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:39.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.124+0000 7f3f9e7fc640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:33:39.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.124+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f8c004eb0 con 0x7f3fa813f710 2024-03-25T19:33:39.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.128+0000 7f3f9e7fc640 10 client.?.objecter ms_handle_connect 0x7f3fa81430c0 2024-03-25T19:33:39.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.128+0000 7f3f9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:39.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.129+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887441434 0 0) 0x7f3fa80abda0 con 0x7f3fa80a98f0 2024-03-25T19:33:39.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.129+0000 7f3f9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:39.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.129+0000 7f3f9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:39.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.129+0000 7f3f9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:39.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.129+0000 7f3f9e7fc640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:33:39.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.129+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa80abac0 con 0x7f3fa80a98f0 2024-03-25T19:33:39.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.143+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2802276608 0 0) 0x7f3fa80abac0 con 0x7f3fa80a98f0 2024-03-25T19:33:39.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.143+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fa80abda0 con 0x7f3fa80a98f0 2024-03-25T19:33:39.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.143+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3fa40018a0 con 0x7f3fa80a98f0 2024-03-25T19:33:39.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.143+0000 7f3f9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:39.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.143+0000 7f3f9e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:39.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.143+0000 7f3f9e7fc640 10 monclient(hunting): dump: 2024-03-25T19:33:39.142 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:39.142 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:39.142 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:39.142 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:39.143 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:39.143 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:39.143 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:39.143 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:39.143 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:39.143 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:39.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.143+0000 7f3f9e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:39.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.147+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359875650 0 0) 0x7f3fa81380e0 con 0x7f3fa81430c0 2024-03-25T19:33:39.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.147+0000 7f3f9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:39.147 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.147+0000 7f3f9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:39.147 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.147+0000 7f3f9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:39.147 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.147+0000 7f3f9e7fc640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:33:39.147 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.147+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa80abac0 con 0x7f3fa81430c0 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.157+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3505545176 0 0) 0x7f3f8c004eb0 con 0x7f3fa813f710 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.157+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fa81380e0 con 0x7f3fa813f710 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.157+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3f98001c00 con 0x7f3fa813f710 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.157+0000 7f3f9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.157+0000 7f3f9e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.157+0000 7f3f9e7fc640 10 monclient(hunting): dump: 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:39.156 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:39.157 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:39.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4000267726 0 0) 0x7f3fa80abda0 con 0x7f3fa80a98f0 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 >> v1:172.21.15.19:6790/0 conn(0x7f3fa81430c0 legacy=0x7f3fa81e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 >> v1:172.21.15.19:6789/0 conn(0x7f3fa813f710 legacy=0x7f3fa8138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3f9e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3f9e7fc640 1 monclient: found mon.b 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3f9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3f9e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa8137d90 con 0x7f3fa80a98f0 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3f9e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3f9e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3f9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3fafb1c640 5 monclient: authenticate success, global_id 4432 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3fafb1c640 10 monclient: _renew_subs 2024-03-25T19:33:39.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3fafb1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3fafb1c640 1 -- 172.21.15.19:0/2003041943 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa8139240 con 0x7f3fa80a98f0 2024-03-25T19:33:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3fafb1c640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3fafb1c640 10 monclient: _renew_subs 2024-03-25T19:33:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3fafb1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3fafb1c640 1 -- 172.21.15.19:0/2003041943 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa8136830 con 0x7f3fa80a98f0 2024-03-25T19:33:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.163+0000 7f3fafb1c640 1 librados: init done 2024-03-25T19:33:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.165+0000 7f3fafb1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.165+0000 7f3fafb1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.165+0000 7f3fafb1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.165+0000 7f3fafb1c640 1 -- 172.21.15.19:0/2003041943 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa80627b0 con 0x7f3fa80a98f0 2024-03-25T19:33:39.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.438+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fa4001cb0 con 0x7f3fa80a98f0 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.438+0000 7f3f9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.438+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3fa4004980 con 0x7f3fa80a98f0 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.438+0000 7f3f9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.438+0000 7f3f9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.438+0000 7f3f9e7fc640 10 monclient: dump: 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:39.438 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:39.439 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:39.439 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:39.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.438+0000 7f3f9ffff640 4 set_mon_vals no callback set 2024-03-25T19:33:39.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.439+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f3fa4004b60 con 0x7f3fa80a98f0 2024-03-25T19:33:39.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.440+0000 7f3fad0b7640 10 monclient: get_auth_request con 0x7f3f8c06dfb0 auth_method 0 2024-03-25T19:33:39.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.440+0000 7f3f9e7fc640 10 client.4432.objecter ms_handle_connect 0x7f3f8c06dfb0 2024-03-25T19:33:39.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.443+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f3fa4054740 con 0x7f3fa80a98f0 2024-03-25T19:33:39.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.443+0000 7f3f9e7fc640 10 client.4432.objecter ms_dispatch 0x7f3fa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:39.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.443+0000 7f3f9e7fc640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:39.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.443+0000 7f3f9e7fc640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:39.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.444+0000 7f3f9e7fc640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T19:33:39.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.444+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3fa4050080 con 0x7f3fa80a98f0 2024-03-25T19:33:39.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.444+0000 7f3f9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:39.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.444+0000 7f3f9e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:39.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.697+0000 7f3fafb1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:39.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.697+0000 7f3fafb1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:39.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.697+0000 7f3fafb1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:39.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.697+0000 7f3fafb1c640 1 -- 172.21.15.19:0/2003041943 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fa813f070 con 0x7f3fa80a98f0 2024-03-25T19:33:39.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.731+0000 7f3f9e7fc640 1 -- 172.21.15.19:0/2003041943 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f3fa4052180 con 0x7f3fa80a98f0 2024-03-25T19:33:39.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.731+0000 7f3f9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:39.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.731+0000 7f3f9e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:39.730 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:33:39.730 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:39.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 10 librados: watch_flush enter 2024-03-25T19:33:39.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 10 librados: watch_flush exit 2024-03-25T19:33:39.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T19:33:39.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 10 client.4432.objecter successfully canceled tick 2024-03-25T19:33:39.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 1 -- 172.21.15.19:0/2003041943 >> v1:172.21.15.188:6800/2207506229 conn(0x7f3f8c06dfb0 legacy=0x7f3f8c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:39.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 10 monclient: shutdown 2024-03-25T19:33:39.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:39.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 1 -- 172.21.15.19:0/2003041943 >> v1:172.21.15.188:6789/0 conn(0x7f3fa80a98f0 legacy=0x7f3fa80ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:39.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 1 -- 172.21.15.19:0/2003041943 shutdown_connections 2024-03-25T19:33:39.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 1 -- 172.21.15.19:0/2003041943 >> 172.21.15.19:0/2003041943 conn(0x7f3fa80b7ea0 msgr2=0x7f3fa8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:39.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 1 -- 172.21.15.19:0/2003041943 shutdown_connections 2024-03-25T19:33:39.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 1 -- 172.21.15.19:0/2003041943 wait complete. 2024-03-25T19:33:39.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:39.733+0000 7f3fafb1c640 1 librados: shutdown 2024-03-25T19:33:39.744 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:46.550 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:46.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.670+0000 7f50ee80e640 10 monclient: get_monmap_and_config 2024-03-25T19:33:46.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.670+0000 7f50ee80e640 10 monclient: build_initial_monmap 2024-03-25T19:33:46.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.670+0000 7f50ee80e640 10 monclient: monmap: 2024-03-25T19:33:46.669 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:46.669 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:46.669 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:33:46.670489+0000 2024-03-25T19:33:46.669 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:33:46.670489+0000 2024-03-25T19:33:46.669 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:46.669 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:46.669 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:46.669 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:33:46.669 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:33:46.670 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:46.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.671+0000 7f50ee80e640 1 Processor -- start 2024-03-25T19:33:46.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.671+0000 7f50ee80e640 1 -- start start 2024-03-25T19:33:46.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.671+0000 7f50ee80e640 10 monclient: init 2024-03-25T19:33:46.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.672+0000 7f50ee80e640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:46.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.672+0000 7f50ee80e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:46.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.672+0000 7f50ee80e640 10 monclient(hunting): picked mon.noname-b con 0x7f50e80b4f10 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:46.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.672+0000 7f50ee80e640 10 monclient(hunting): picked mon.noname-a con 0x7f50e80b44d0 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:46.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.672+0000 7f50ee80e640 10 monclient(hunting): picked mon.noname-c con 0x7f50e813f740 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:46.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.672+0000 7f50ee80e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e80b0140 con 0x7f50e80b44d0 2024-03-25T19:33:46.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.672+0000 7f50ee80e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e80af3a0 con 0x7f50e813f740 2024-03-25T19:33:46.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.672+0000 7f50ee80e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e80af530 con 0x7f50e80b4f10 2024-03-25T19:33:46.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.672+0000 7f50ee80e640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:46.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.672+0000 7f50ee80e640 10 monclient(hunting): authenticate will time out at 6279.941283s 2024-03-25T19:33:46.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.683+0000 7f50e77fe640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f50e80b44d0 0x7f50e80b16c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:35100/0 (socket says 172.21.15.19:35100) 2024-03-25T19:33:46.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.683+0000 7f50e77fe640 1 -- 172.21.15.19:0/1156736342 learned_addr learned my addr 172.21.15.19:0/1156736342 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:46.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.695+0000 7f50e6ffd640 1 -- 172.21.15.19:0/1156736342 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951689032 0 0) 0x7f50e80b0140 con 0x7f50e80b44d0 2024-03-25T19:33:46.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.695+0000 7f50e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:46.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.695+0000 7f50e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:46.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.695+0000 7f50e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:46.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.695+0000 7f50e6ffd640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:33:46.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.695+0000 7f50e6ffd640 1 -- 172.21.15.19:0/1156736342 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d0003520 con 0x7f50e80b44d0 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.697+0000 7f50e6ffd640 1 -- 172.21.15.19:0/1156736342 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487035184 0 0) 0x7f50e80af3a0 con 0x7f50e813f740 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.697+0000 7f50e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.697+0000 7f50e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.697+0000 7f50e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.697+0000 7f50e6ffd640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.697+0000 7f50e6ffd640 1 -- 172.21.15.19:0/1156736342 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e80b0140 con 0x7f50e813f740 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 1 -- 172.21.15.19:0/1156736342 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2911904420 0 0) 0x7f50e80b0140 con 0x7f50e813f740 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 1 -- 172.21.15.19:0/1156736342 >> v1:172.21.15.19:6790/0 conn(0x7f50e80b4f10 legacy=0x7f50e80ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 1 -- 172.21.15.19:0/1156736342 >> v1:172.21.15.19:6789/0 conn(0x7f50e80b44d0 legacy=0x7f50e80b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:33:46.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 1 -- 172.21.15.19:0/1156736342 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e80afcd0 con 0x7f50e813f740 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 1 -- 172.21.15.19:0/1156736342 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f50e00019f0 con 0x7f50e813f740 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50e6ffd640 10 monclient: dump: 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:46.732 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50ee80e640 5 monclient: authenticate success, global_id 4435 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.703+0000 7f50ee80e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.718+0000 7f50e6ffd640 1 -- 172.21.15.19:0/1156736342 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50e0001b50 con 0x7f50e813f740 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.718+0000 7f50e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.718+0000 7f50e6ffd640 1 -- 172.21.15.19:0/1156736342 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f50e0001f20 con 0x7f50e813f740 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.718+0000 7f50ee80e640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.718+0000 7f50ee80e640 4 set_mon_vals no callback set 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.718+0000 7f50e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.718+0000 7f50e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.718+0000 7f50e6ffd640 10 monclient: dump: 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:46.733 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.719+0000 7f50ee80e640 10 monclient: shutdown 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.719+0000 7f50ee80e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.719+0000 7f50ee80e640 1 -- 172.21.15.19:0/1156736342 >> v1:172.21.15.188:6789/0 conn(0x7f50e813f740 legacy=0x7f50e813fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.719+0000 7f50ee80e640 1 -- 172.21.15.19:0/1156736342 shutdown_connections 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.719+0000 7f50ee80e640 1 -- 172.21.15.19:0/1156736342 >> 172.21.15.19:0/1156736342 conn(0x7f50e80617a0 msgr2=0x7f50e8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.719+0000 7f50ee80e640 1 -- 172.21.15.19:0/1156736342 shutdown_connections 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.719+0000 7f50ee80e640 1 -- 172.21.15.19:0/1156736342 wait complete. 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.720+0000 7f50ee80e640 10 monclient: build_initial_monmap 2024-03-25T19:33:46.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.720+0000 7f50ee80e640 10 monclient: monmap: 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.720+0000 7f50ee80e640 1 librados: starting msgr at 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.720+0000 7f50ee80e640 1 librados: starting objecter 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.720+0000 7f50ee80e640 1 Processor -- start 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.720+0000 7f50ee80e640 1 -- start start 2024-03-25T19:33:46.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.720+0000 7f50ee80e640 1 librados: setting wanted keys 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.720+0000 7f50ee80e640 1 librados: calling monclient init 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.720+0000 7f50ee80e640 10 monclient: init 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.721+0000 7f50ee80e640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.721+0000 7f50ee80e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.721+0000 7f50ee80e640 10 monclient(hunting): picked mon.noname-a con 0x7f50e80b44d0 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.721+0000 7f50ee80e640 10 monclient(hunting): picked mon.noname-b con 0x7f50e80b4f10 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.721+0000 7f50ee80e640 10 monclient(hunting): picked mon.noname-c con 0x7f50e813f740 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.721+0000 7f50ee80e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e80a9850 con 0x7f50e80b44d0 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.721+0000 7f50ee80e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e80a9470 con 0x7f50e80b4f10 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.721+0000 7f50ee80e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e80a9600 con 0x7f50e813f740 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.721+0000 7f50ee80e640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.721+0000 7f50ee80e640 10 monclient(hunting): authenticate will time out at 6279.990523s 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.729+0000 7f50e7fff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f50e80b44d0 0x7f50e80ac810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:35148/0 (socket says 172.21.15.19:35148) 2024-03-25T19:33:46.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.729+0000 7f50e7fff640 1 -- 172.21.15.19:0/1455482782 learned_addr learned my addr 172.21.15.19:0/1455482782 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:46.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.732+0000 7f50e4ff9640 10 client.?.objecter ms_handle_connect 0x7f50e80b44d0 2024-03-25T19:33:46.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.732+0000 7f50e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:46.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.739+0000 7f50e4ff9640 10 client.?.objecter ms_handle_connect 0x7f50e80b4f10 2024-03-25T19:33:46.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.739+0000 7f50e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:46.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.740+0000 7f50e4ff9640 10 client.?.objecter ms_handle_connect 0x7f50e813f740 2024-03-25T19:33:46.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.740+0000 7f50e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:46.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.745+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798610533 0 0) 0x7f50e80a9850 con 0x7f50e80b44d0 2024-03-25T19:33:46.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.745+0000 7f50e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:46.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.745+0000 7f50e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:46.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.745+0000 7f50e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:46.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.745+0000 7f50e4ff9640 10 monclient(hunting): my global_id is 4446 2024-03-25T19:33:46.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.745+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d4004eb0 con 0x7f50e80b44d0 2024-03-25T19:33:46.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.750+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806133276 0 0) 0x7f50e80a9470 con 0x7f50e80b4f10 2024-03-25T19:33:46.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.750+0000 7f50e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:46.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.750+0000 7f50e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:46.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.750+0000 7f50e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:46.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.750+0000 7f50e4ff9640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:33:46.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.750+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e80a9850 con 0x7f50e80b4f10 2024-03-25T19:33:46.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.751+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076787725 0 0) 0x7f50e80a9600 con 0x7f50e813f740 2024-03-25T19:33:46.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.751+0000 7f50e4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:46.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.751+0000 7f50e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:46.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.751+0000 7f50e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:46.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.751+0000 7f50e4ff9640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:33:46.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.751+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e80a9470 con 0x7f50e813f740 2024-03-25T19:33:46.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.760+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2574619650 0 0) 0x7f50d4004eb0 con 0x7f50e80b44d0 2024-03-25T19:33:46.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.760+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50e80a9600 con 0x7f50e80b44d0 2024-03-25T19:33:46.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.760+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f50d80019d0 con 0x7f50e80b44d0 2024-03-25T19:33:46.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.760+0000 7f50e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:46.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.760+0000 7f50e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:46.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.760+0000 7f50e4ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:46.760 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:46.760 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:46.760 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:46.760 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:46.760 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:46.761 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:46.761 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:46.761 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:46.761 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:46.761 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:46.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.760+0000 7f50e4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:46.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.767+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1505084070 0 0) 0x7f50e80a9850 con 0x7f50e80b4f10 2024-03-25T19:33:46.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.767+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50d4004eb0 con 0x7f50e80b4f10 2024-03-25T19:33:46.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.767+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f50dc0018d0 con 0x7f50e80b4f10 2024-03-25T19:33:46.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.767+0000 7f50e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.767+0000 7f50e4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.767+0000 7f50e4ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:46.768 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:46.768 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:46.768 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:46.768 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:46.768 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:46.768 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:46.768 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:46.768 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:46.768 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:46.769 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:46.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3041306963 0 0) 0x7f50e80a9470 con 0x7f50e813f740 2024-03-25T19:33:46.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50e80a9850 con 0x7f50e813f740 2024-03-25T19:33:46.772 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f50e0001e60 con 0x7f50e813f740 2024-03-25T19:33:46.772 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:46.772 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:46.772 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:46.772 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:46.772 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:46.772 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:46.772 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:46.773 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:46.773 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:46.773 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:46.773 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:46.773 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:46.773 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:46.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1414608128 0 0) 0x7f50e80a9600 con 0x7f50e80b44d0 2024-03-25T19:33:46.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 >> v1:172.21.15.19:6790/0 conn(0x7f50e813f740 legacy=0x7f50e80aa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 >> v1:172.21.15.188:6789/0 conn(0x7f50e80b4f10 legacy=0x7f50e80a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:33:46.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 1 monclient: found mon.a 2024-03-25T19:33:46.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:46.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:46.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.770+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e80abcb0 con 0x7f50e80b44d0 2024-03-25T19:33:46.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.771+0000 7f50e4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:33:46.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.771+0000 7f50e4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:33:46.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.771+0000 7f50e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:46.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.771+0000 7f50ee80e640 5 monclient: authenticate success, global_id 4446 2024-03-25T19:33:46.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.771+0000 7f50ee80e640 10 monclient: _renew_subs 2024-03-25T19:33:46.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.771+0000 7f50ee80e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:46.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.771+0000 7f50ee80e640 1 -- 172.21.15.19:0/1455482782 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e80abf60 con 0x7f50e80b44d0 2024-03-25T19:33:46.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.771+0000 7f50ee80e640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:46.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.771+0000 7f50ee80e640 10 monclient: _renew_subs 2024-03-25T19:33:46.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.771+0000 7f50ee80e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:46.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.771+0000 7f50ee80e640 1 -- 172.21.15.19:0/1455482782 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e81e3010 con 0x7f50e80b44d0 2024-03-25T19:33:46.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.771+0000 7f50ee80e640 1 librados: init done 2024-03-25T19:33:46.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.772+0000 7f50ee80e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:46.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.772+0000 7f50ee80e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:46.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.772+0000 7f50ee80e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:46.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:46.772+0000 7f50ee80e640 1 -- 172.21.15.19:0/1455482782 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e80627b0 con 0x7f50e80b44d0 2024-03-25T19:33:47.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.082+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50d8003610 con 0x7f50e80b44d0 2024-03-25T19:33:47.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.082+0000 7f50e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:47.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.082+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f50d8003a20 con 0x7f50e80b44d0 2024-03-25T19:33:47.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.082+0000 7f50e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:47.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.082+0000 7f50e4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:47.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.082+0000 7f50e4ff9640 10 monclient: dump: 2024-03-25T19:33:47.082 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:47.082 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:47.082 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:47.082 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:47.082 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:47.082 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:47.082 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:47.082 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:47.083 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:47.083 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:47.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.082+0000 7f50e67fc640 4 set_mon_vals no callback set 2024-03-25T19:33:47.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.083+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f50d8003c40 con 0x7f50e80b44d0 2024-03-25T19:33:47.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.083+0000 7f50e77fe640 10 monclient: get_auth_request con 0x7f50d406e0f0 auth_method 0 2024-03-25T19:33:47.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.084+0000 7f50e4ff9640 10 client.4446.objecter ms_handle_connect 0x7f50d406e0f0 2024-03-25T19:33:47.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.089+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f50d8054820 con 0x7f50e80b44d0 2024-03-25T19:33:47.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.089+0000 7f50e4ff9640 10 client.4446.objecter ms_dispatch 0x7f50e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:47.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.089+0000 7f50e4ff9640 3 client.4446.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:47.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.089+0000 7f50e4ff9640 3 client.4446.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:47.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.089+0000 7f50e4ff9640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-25T19:33:47.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.089+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50d8050330 con 0x7f50e80b44d0 2024-03-25T19:33:47.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.089+0000 7f50e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:47.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.089+0000 7f50e4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:47.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.311+0000 7f50ee80e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:47.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.311+0000 7f50ee80e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:47.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.312+0000 7f50ee80e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:33:47.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.312+0000 7f50ee80e640 1 -- 172.21.15.19:0/1455482782 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50e80ac0f0 con 0x7f50e80b44d0 2024-03-25T19:33:47.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.335+0000 7f50e4ff9640 1 -- 172.21.15.19:0/1455482782 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f50d8052260 con 0x7f50e80b44d0 2024-03-25T19:33:47.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.335+0000 7f50e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:47.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.335+0000 7f50e4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:47.334 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:33:47.334 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:47.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.337+0000 7f50ee80e640 10 librados: watch_flush enter 2024-03-25T19:33:47.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.337+0000 7f50ee80e640 10 librados: watch_flush exit 2024-03-25T19:33:47.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.337+0000 7f50ee80e640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-25T19:33:47.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.337+0000 7f50ee80e640 10 client.4446.objecter successfully canceled tick 2024-03-25T19:33:47.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.337+0000 7f50ee80e640 1 -- 172.21.15.19:0/1455482782 >> v1:172.21.15.188:6800/2207506229 conn(0x7f50d406e0f0 legacy=0x7f50d40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:47.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.337+0000 7f50ee80e640 10 monclient: shutdown 2024-03-25T19:33:47.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.337+0000 7f50ee80e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:47.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.337+0000 7f50ee80e640 1 -- 172.21.15.19:0/1455482782 >> v1:172.21.15.19:6789/0 conn(0x7f50e80b44d0 legacy=0x7f50e80ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:47.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.337+0000 7f50ee80e640 1 -- 172.21.15.19:0/1455482782 shutdown_connections 2024-03-25T19:33:47.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.337+0000 7f50ee80e640 1 -- 172.21.15.19:0/1455482782 >> 172.21.15.19:0/1455482782 conn(0x7f50e80617a0 msgr2=0x7f50e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:47.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.337+0000 7f50ee80e640 1 -- 172.21.15.19:0/1455482782 shutdown_connections 2024-03-25T19:33:47.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.337+0000 7f50ee80e640 1 -- 172.21.15.19:0/1455482782 wait complete. 2024-03-25T19:33:47.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:47.338+0000 7f50ee80e640 1 librados: shutdown 2024-03-25T19:33:47.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:52.485 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:15.214 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.499 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:15.228 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.525 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:15.255 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.547 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:05:15.250 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.549 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:15.279 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.553 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:05:15.256 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.575 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:15.304 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.588 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:05:15.291 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.608 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:15.337 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:54.156 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.271+0000 7fc8e0bb6640 10 monclient: get_monmap_and_config 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.271+0000 7fc8e0bb6640 10 monclient: build_initial_monmap 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.271+0000 7fc8e0bb6640 10 monclient: monmap: 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:33:54.272371+0000 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:33:54.272371+0000 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:33:54.271 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.271+0000 7fc8e0bb6640 1 Processor -- start 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.271+0000 7fc8e0bb6640 1 -- start start 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.271+0000 7fc8e0bb6640 10 monclient: init 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.272+0000 7fc8e0bb6640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.272+0000 7fc8e0bb6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.272+0000 7fc8e0bb6640 10 monclient(hunting): picked mon.noname-a con 0x7fc8dc0b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.272+0000 7fc8e0bb6640 10 monclient(hunting): picked mon.noname-b con 0x7fc8dc0b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.272+0000 7fc8e0bb6640 10 monclient(hunting): picked mon.noname-c con 0x7fc8dc14be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.272+0000 7fc8e0bb6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc0af940 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.272+0000 7fc8e0bb6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc0b0a70 con 0x7fc8dc14be90 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.272+0000 7fc8e0bb6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc0b0d50 con 0x7fc8dc0b1730 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.272+0000 7fc8e0bb6640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:54.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.272+0000 7fc8e0bb6640 10 monclient(hunting): authenticate will time out at 6287.542530s 2024-03-25T19:33:54.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.284+0000 7fc8dad9d640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fc8dc14be90 0x7fc8dc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:58510/0 (socket says 172.21.15.19:58510) 2024-03-25T19:33:54.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.284+0000 7fc8dad9d640 1 -- 172.21.15.19:0/2708593801 learned_addr learned my addr 172.21.15.19:0/2708593801 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:54.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.300+0000 7fc8d959a640 1 -- 172.21.15.19:0/2708593801 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906964514 0 0) 0x7fc8dc0b0a70 con 0x7fc8dc14be90 2024-03-25T19:33:54.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.300+0000 7fc8d959a640 10 monclient(hunting): _negotiate 2024-03-25T19:33:54.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.300+0000 7fc8d959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:54.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.300+0000 7fc8d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:54.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.300+0000 7fc8d959a640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:33:54.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.300+0000 7fc8d959a640 1 -- 172.21.15.19:0/2708593801 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8c4003520 con 0x7fc8dc14be90 2024-03-25T19:33:54.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.301+0000 7fc8d959a640 1 -- 172.21.15.19:0/2708593801 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692931212 0 0) 0x7fc8dc0b0d50 con 0x7fc8dc0b1730 2024-03-25T19:33:54.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.301+0000 7fc8d959a640 10 monclient(hunting): _negotiate 2024-03-25T19:33:54.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.301+0000 7fc8d959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:54.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.301+0000 7fc8d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:54.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.301+0000 7fc8d959a640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:33:54.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.301+0000 7fc8d959a640 1 -- 172.21.15.19:0/2708593801 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8dc0b0a70 con 0x7fc8dc0b1730 2024-03-25T19:33:54.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 1 -- 172.21.15.19:0/2708593801 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3191578035 0 0) 0x7fc8c4003520 con 0x7fc8dc14be90 2024-03-25T19:33:54.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 1 -- 172.21.15.19:0/2708593801 >> v1:172.21.15.19:6790/0 conn(0x7fc8dc0b1730 legacy=0x7fc8dc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 1 -- 172.21.15.19:0/2708593801 >> v1:172.21.15.19:6789/0 conn(0x7fc8dc0b4d60 legacy=0x7fc8dc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 10 monclient: _finish_hunting 0 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 1 monclient: found mon.noname-c 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 1 -- 172.21.15.19:0/2708593801 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc0b1030 con 0x7fc8dc14be90 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 10 monclient: _finish_auth 0 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 10 monclient: _check_auth_tickets 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 1 -- 172.21.15.19:0/2708593801 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc8d4001a10 con 0x7fc8dc14be90 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8e0bb6640 5 monclient: authenticate success, global_id 4441 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8e0bb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:54.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8d959a640 10 monclient: dump: 2024-03-25T19:33:54.316 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:54.316 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:54.316 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:54.316 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:54.316 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:54.316 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:54.316 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:54.316 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:54.316 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:54.316 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:54.316 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.314+0000 7fc8e0bb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:54.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.325+0000 7fc8d959a640 1 -- 172.21.15.19:0/2708593801 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8d4001b70 con 0x7fc8dc14be90 2024-03-25T19:33:54.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8d959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:54.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8d959a640 1 -- 172.21.15.19:0/2708593801 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc8d4001f80 con 0x7fc8dc14be90 2024-03-25T19:33:54.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8e0bb6640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:54.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8e0bb6640 4 set_mon_vals no callback set 2024-03-25T19:33:54.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:54.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8d959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:54.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8d959a640 10 monclient: dump: 2024-03-25T19:33:54.326 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:54.326 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:54.326 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:54.326 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:54.327 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:54.327 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:54.327 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:54.327 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:54.327 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:54.327 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:54.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8e0bb6640 10 monclient: shutdown 2024-03-25T19:33:54.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8e0bb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:54.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2708593801 >> v1:172.21.15.188:6789/0 conn(0x7fc8dc14be90 legacy=0x7fc8dc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:54.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2708593801 shutdown_connections 2024-03-25T19:33:54.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2708593801 >> 172.21.15.19:0/2708593801 conn(0x7fc8dc0617a0 msgr2=0x7fc8dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:54.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.326+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2708593801 shutdown_connections 2024-03-25T19:33:54.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.327+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2708593801 wait complete. 2024-03-25T19:33:54.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.328+0000 7fc8e0bb6640 10 monclient: build_initial_monmap 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.328+0000 7fc8e0bb6640 10 monclient: monmap: 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.328+0000 7fc8e0bb6640 1 librados: starting msgr at 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.328+0000 7fc8e0bb6640 1 librados: starting objecter 2024-03-25T19:33:54.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.328+0000 7fc8e0bb6640 1 Processor -- start 2024-03-25T19:33:54.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.328+0000 7fc8e0bb6640 1 -- start start 2024-03-25T19:33:54.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.328+0000 7fc8e0bb6640 1 librados: setting wanted keys 2024-03-25T19:33:54.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.328+0000 7fc8e0bb6640 1 librados: calling monclient init 2024-03-25T19:33:54.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.328+0000 7fc8e0bb6640 10 monclient: init 2024-03-25T19:33:54.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.329+0000 7fc8e0bb6640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:54.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.329+0000 7fc8e0bb6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:54.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.329+0000 7fc8e0bb6640 10 monclient(hunting): picked mon.noname-c con 0x7fc8dc0b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:33:54.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.329+0000 7fc8e0bb6640 10 monclient(hunting): picked mon.noname-b con 0x7fc8dc0b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:33:54.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.329+0000 7fc8e0bb6640 10 monclient(hunting): picked mon.noname-a con 0x7fc8dc14be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:33:54.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.329+0000 7fc8e0bb6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc13b350 con 0x7fc8dc14be90 2024-03-25T19:33:54.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.329+0000 7fc8e0bb6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc13b7e0 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.329+0000 7fc8e0bb6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc13a1e0 con 0x7fc8dc0b1730 2024-03-25T19:33:54.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.329+0000 7fc8e0bb6640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:54.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.329+0000 7fc8e0bb6640 10 monclient(hunting): authenticate will time out at 6287.599351s 2024-03-25T19:33:54.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.335+0000 7fc8d9d9b640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fc8dc0b4d60 0x7fc8dc136f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:58550/0 (socket says 172.21.15.19:58550) 2024-03-25T19:33:54.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.336+0000 7fc8d9d9b640 1 -- 172.21.15.19:0/2456767452 learned_addr learned my addr 172.21.15.19:0/2456767452 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:33:54.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.337+0000 7fc8caffd640 10 client.?.objecter ms_handle_connect 0x7fc8dc0b4d60 2024-03-25T19:33:54.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.337+0000 7fc8caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:54.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.344+0000 7fc8caffd640 10 client.?.objecter ms_handle_connect 0x7fc8dc0b1730 2024-03-25T19:33:54.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.344+0000 7fc8caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:54.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.350+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799194184 0 0) 0x7fc8dc13b7e0 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.350+0000 7fc8caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:54.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.350+0000 7fc8caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:54.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.350+0000 7fc8caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:54.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.350+0000 7fc8caffd640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:33:54.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.350+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8bc004eb0 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.351+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844580640 0 0) 0x7fc8dc13a1e0 con 0x7fc8dc0b1730 2024-03-25T19:33:54.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.351+0000 7fc8caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:54.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.351+0000 7fc8caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:54.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.351+0000 7fc8caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:54.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.351+0000 7fc8caffd640 10 monclient(hunting): my global_id is 4424 2024-03-25T19:33:54.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.351+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8dc13b7e0 con 0x7fc8dc0b1730 2024-03-25T19:33:54.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.351+0000 7fc8caffd640 10 client.?.objecter ms_handle_connect 0x7fc8dc14be90 2024-03-25T19:33:54.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.351+0000 7fc8caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.372+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1452113898 0 0) 0x7fc8bc004eb0 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.373+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8dc13a1e0 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.373+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc8d0001a10 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.373+0000 7fc8caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.373+0000 7fc8caffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.373+0000 7fc8caffd640 10 monclient(hunting): dump: 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:54.373 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:54.374 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:54.374 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:54.374 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:54.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.373+0000 7fc8caffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.380+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678480430 0 0) 0x7fc8dc13b350 con 0x7fc8dc14be90 2024-03-25T19:33:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.380+0000 7fc8caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.380+0000 7fc8caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.380+0000 7fc8caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.380+0000 7fc8caffd640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:33:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.380+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8bc004eb0 con 0x7fc8dc14be90 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.383+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1826804053 0 0) 0x7fc8dc13b7e0 con 0x7fc8dc0b1730 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.383+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8dc13b350 con 0x7fc8dc0b1730 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.383+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc8cc001830 con 0x7fc8dc0b1730 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.383+0000 7fc8caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.383+0000 7fc8caffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.383+0000 7fc8caffd640 10 monclient(hunting): dump: 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:54.384 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:54.385 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:54.385 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 93936897 0 0) 0x7fc8dc13a1e0 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 >> v1:172.21.15.19:6790/0 conn(0x7fc8dc0b1730 legacy=0x7fc8dc13a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 >> v1:172.21.15.19:6789/0 conn(0x7fc8dc14be90 legacy=0x7fc8dc1d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8caffd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8caffd640 1 monclient: found mon.b 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8caffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc137670 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8caffd640 10 monclient: _finish_auth 0 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8caffd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8e0bb6640 5 monclient: authenticate success, global_id 4444 2024-03-25T19:33:54.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8e0bb6640 10 monclient: _renew_subs 2024-03-25T19:33:54.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8e0bb6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:54.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2456767452 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc137920 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8e0bb6640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:54.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8e0bb6640 10 monclient: _renew_subs 2024-03-25T19:33:54.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8e0bb6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:54.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2456767452 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc1428a0 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.396+0000 7fc8e0bb6640 1 librados: init done 2024-03-25T19:33:54.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.397+0000 7fc8e0bb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:54.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.397+0000 7fc8e0bb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:54.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.397+0000 7fc8e0bb6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:54.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.397+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2456767452 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc139f90 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.674+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8d0001c50 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.674+0000 7fc8caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:54.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.674+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc8d0003a60 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.674+0000 7fc8caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:54.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.674+0000 7fc8caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:54.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.674+0000 7fc8caffd640 10 monclient: dump: 2024-03-25T19:33:54.674 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:33:54.675 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:33:54.675 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:54.675 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:33:54.675 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:54.675 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:33:54.675 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:33:54.675 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:33:54.675 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:33:54.675 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:33:54.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.674+0000 7fc8d8d99640 4 set_mon_vals no callback set 2024-03-25T19:33:54.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.675+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fc8d0003c40 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.676+0000 7fc8da59c640 10 monclient: get_auth_request con 0x7fc8bc06dfb0 auth_method 0 2024-03-25T19:33:54.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.676+0000 7fc8caffd640 10 client.4444.objecter ms_handle_connect 0x7fc8bc06dfb0 2024-03-25T19:33:54.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.681+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fc8d00548c0 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.681+0000 7fc8caffd640 10 client.4444.objecter ms_dispatch 0x7fc8dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:54.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.681+0000 7fc8caffd640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:54.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.681+0000 7fc8caffd640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:54.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.681+0000 7fc8caffd640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-25T19:33:54.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.681+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8d00503d0 con 0x7fc8dc0b4d60 2024-03-25T19:33:54.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.681+0000 7fc8caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:54.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.681+0000 7fc8caffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:54.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.924+0000 7fc8e0bb6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:54.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.924+0000 7fc8e0bb6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:54.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.924+0000 7fc8e0bb6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:33:54.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:54.924+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2456767452 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8dc0627b0 con 0x7fc8dc0b4d60 2024-03-25T19:33:55.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.054+0000 7fc8caffd640 1 -- 172.21.15.19:0/2456767452 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fc8d0052300 con 0x7fc8dc0b4d60 2024-03-25T19:33:55.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.054+0000 7fc8caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:55.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.054+0000 7fc8caffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:55.055 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:33:55.055 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:55.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 10 librados: watch_flush enter 2024-03-25T19:33:55.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 10 librados: watch_flush exit 2024-03-25T19:33:55.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-25T19:33:55.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 10 client.4444.objecter successfully canceled tick 2024-03-25T19:33:55.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2456767452 >> v1:172.21.15.188:6800/2207506229 conn(0x7fc8bc06dfb0 legacy=0x7fc8bc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:55.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 10 monclient: shutdown 2024-03-25T19:33:55.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:55.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2456767452 >> v1:172.21.15.188:6789/0 conn(0x7fc8dc0b4d60 legacy=0x7fc8dc136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:55.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2456767452 shutdown_connections 2024-03-25T19:33:55.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2456767452 >> 172.21.15.19:0/2456767452 conn(0x7fc8dc0617a0 msgr2=0x7fc8dc145110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:55.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2456767452 shutdown_connections 2024-03-25T19:33:55.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 1 -- 172.21.15.19:0/2456767452 wait complete. 2024-03-25T19:33:55.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:33:55.057+0000 7fc8e0bb6640 1 librados: shutdown 2024-03-25T19:33:55.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:01.876 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:01.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.974+0000 7fe3fae5c640 10 monclient: get_monmap_and_config 2024-03-25T19:34:01.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.974+0000 7fe3fae5c640 10 monclient: build_initial_monmap 2024-03-25T19:34:01.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.975+0000 7fe3fae5c640 10 monclient: monmap: 2024-03-25T19:34:01.975 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:01.976 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:01.976 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:34:01.976245+0000 2024-03-25T19:34:01.976 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:34:01.976245+0000 2024-03-25T19:34:01.976 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:01.976 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:01.976 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:01.976 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:34:01.976 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:34:01.976 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:01.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.975+0000 7fe3fae5c640 1 Processor -- start 2024-03-25T19:34:01.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.976+0000 7fe3fae5c640 1 -- start start 2024-03-25T19:34:01.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.976+0000 7fe3fae5c640 10 monclient: init 2024-03-25T19:34:01.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.976+0000 7fe3fae5c640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:01.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.976+0000 7fe3fae5c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:01.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.976+0000 7fe3fae5c640 10 monclient(hunting): picked mon.noname-a con 0x7fe3f40a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:01.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.976+0000 7fe3fae5c640 10 monclient(hunting): picked mon.noname-c con 0x7fe3f413f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:01.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.976+0000 7fe3fae5c640 10 monclient(hunting): picked mon.noname-b con 0x7fe3f41430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:01.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.976+0000 7fe3fae5c640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f4142520 con 0x7fe3f40a98f0 2024-03-25T19:34:01.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.976+0000 7fe3fae5c640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f40b28a0 con 0x7fe3f413f710 2024-03-25T19:34:01.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.976+0000 7fe3fae5c640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f40b2a30 con 0x7fe3f41430c0 2024-03-25T19:34:01.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.976+0000 7fe3fae5c640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:01.979 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.976+0000 7fe3fae5c640 10 monclient(hunting): authenticate will time out at 6295.246862s 2024-03-25T19:34:01.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.985+0000 7fe3f8bf8640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fe3f40a98f0 0x7fe3f40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:51034/0 (socket says 172.21.15.19:51034) 2024-03-25T19:34:01.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.985+0000 7fe3f8bf8640 1 -- 172.21.15.19:0/2225508868 learned_addr learned my addr 172.21.15.19:0/2225508868 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:01.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.997+0000 7fe3f37fe640 1 -- 172.21.15.19:0/2225508868 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632504291 0 0) 0x7fe3f4142520 con 0x7fe3f40a98f0 2024-03-25T19:34:01.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.997+0000 7fe3f37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:01.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.997+0000 7fe3f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:01.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.997+0000 7fe3f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:01.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.997+0000 7fe3f37fe640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:34:01.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:01.997+0000 7fe3f37fe640 1 -- 172.21.15.19:0/2225508868 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e4003520 con 0x7fe3f40a98f0 2024-03-25T19:34:02.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.002+0000 7fe3f37fe640 1 -- 172.21.15.19:0/2225508868 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406548410 0 0) 0x7fe3f40b28a0 con 0x7fe3f413f710 2024-03-25T19:34:02.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.002+0000 7fe3f37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:02.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.002+0000 7fe3f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:02.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.002+0000 7fe3f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:02.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.002+0000 7fe3f37fe640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:34:02.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.002+0000 7fe3f37fe640 1 -- 172.21.15.19:0/2225508868 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f4142520 con 0x7fe3f413f710 2024-03-25T19:34:02.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 1 -- 172.21.15.19:0/2225508868 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4214121693 0 0) 0x7fe3e4003520 con 0x7fe3f40a98f0 2024-03-25T19:34:02.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 1 -- 172.21.15.19:0/2225508868 >> v1:172.21.15.19:6790/0 conn(0x7fe3f41430c0 legacy=0x7fe3f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 1 -- 172.21.15.19:0/2225508868 >> v1:172.21.15.188:6789/0 conn(0x7fe3f413f710 legacy=0x7fe3f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:02.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 1 monclient: found mon.noname-a 2024-03-25T19:34:02.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:02.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:34:02.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 1 -- 172.21.15.19:0/2225508868 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f40b44c0 con 0x7fe3f40a98f0 2024-03-25T19:34:02.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:02.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:02.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:02.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 1 -- 172.21.15.19:0/2225508868 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe3e8001690 con 0x7fe3f40a98f0 2024-03-25T19:34:02.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3fae5c640 5 monclient: authenticate success, global_id 4452 2024-03-25T19:34:02.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3fae5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:02.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:02.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:02.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3f37fe640 10 monclient: dump: 2024-03-25T19:34:02.020 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:02.020 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:02.020 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:02.021 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:02.021 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:02.021 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:02.021 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:02.021 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:02.021 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:02.021 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:02.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.019+0000 7fe3fae5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:02.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.027+0000 7fe3f37fe640 1 -- 172.21.15.19:0/2225508868 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3e80017f0 con 0x7fe3f40a98f0 2024-03-25T19:34:02.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.027+0000 7fe3f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:02.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.027+0000 7fe3f37fe640 1 -- 172.21.15.19:0/2225508868 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe3e8001c00 con 0x7fe3f40a98f0 2024-03-25T19:34:02.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.027+0000 7fe3f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:02.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.027+0000 7fe3f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:02.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.027+0000 7fe3f37fe640 10 monclient: dump: 2024-03-25T19:34:02.027 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:02.027 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:02.027 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:02.027 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:02.027 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:02.027 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:02.027 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:02.028 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:02.028 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:02.028 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:02.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.027+0000 7fe3fae5c640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:02.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.027+0000 7fe3fae5c640 4 set_mon_vals no callback set 2024-03-25T19:34:02.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.027+0000 7fe3fae5c640 10 monclient: shutdown 2024-03-25T19:34:02.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.027+0000 7fe3fae5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:02.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.027+0000 7fe3fae5c640 1 -- 172.21.15.19:0/2225508868 >> v1:172.21.15.19:6789/0 conn(0x7fe3f40a98f0 legacy=0x7fe3f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.028+0000 7fe3fae5c640 1 -- 172.21.15.19:0/2225508868 shutdown_connections 2024-03-25T19:34:02.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.028+0000 7fe3fae5c640 1 -- 172.21.15.19:0/2225508868 >> 172.21.15.19:0/2225508868 conn(0x7fe3f40b7ea0 msgr2=0x7fe3f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:02.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.028+0000 7fe3fae5c640 1 -- 172.21.15.19:0/2225508868 shutdown_connections 2024-03-25T19:34:02.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.028+0000 7fe3fae5c640 1 -- 172.21.15.19:0/2225508868 wait complete. 2024-03-25T19:34:02.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.028+0000 7fe3fae5c640 10 monclient: build_initial_monmap 2024-03-25T19:34:02.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.028+0000 7fe3fae5c640 10 monclient: monmap: 2024-03-25T19:34:02.030 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:02.030 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:02.030 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:34:02.030 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:34:02.030 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:02.030 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:34:02.030 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:02.031 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:34:02.031 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:34:02.031 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:02.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.029+0000 7fe3fae5c640 1 librados: starting msgr at 2024-03-25T19:34:02.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.029+0000 7fe3fae5c640 1 librados: starting objecter 2024-03-25T19:34:02.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.029+0000 7fe3fae5c640 1 Processor -- start 2024-03-25T19:34:02.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.029+0000 7fe3fae5c640 1 -- start start 2024-03-25T19:34:02.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.029+0000 7fe3fae5c640 1 librados: setting wanted keys 2024-03-25T19:34:02.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.029+0000 7fe3fae5c640 1 librados: calling monclient init 2024-03-25T19:34:02.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.029+0000 7fe3fae5c640 10 monclient: init 2024-03-25T19:34:02.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.030+0000 7fe3fae5c640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:02.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.030+0000 7fe3fae5c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:02.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.030+0000 7fe3fae5c640 10 monclient(hunting): picked mon.noname-c con 0x7fe3f40a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:02.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.030+0000 7fe3fae5c640 10 monclient(hunting): picked mon.noname-a con 0x7fe3f413f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:02.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.030+0000 7fe3fae5c640 10 monclient(hunting): picked mon.noname-b con 0x7fe3f41430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:02.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.030+0000 7fe3fae5c640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f4132a30 con 0x7fe3f413f710 2024-03-25T19:34:02.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.030+0000 7fe3fae5c640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f4138c80 con 0x7fe3f41430c0 2024-03-25T19:34:02.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.030+0000 7fe3fae5c640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f4138f60 con 0x7fe3f40a98f0 2024-03-25T19:34:02.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.030+0000 7fe3fae5c640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:02.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.030+0000 7fe3fae5c640 10 monclient(hunting): authenticate will time out at 6295.300587s 2024-03-25T19:34:02.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.035+0000 7fe3f93f9640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fe3f41430c0 0x7fe3f4132320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:40988/0 (socket says 172.21.15.19:40988) 2024-03-25T19:34:02.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.035+0000 7fe3f93f9640 1 -- 172.21.15.19:0/1496775600 learned_addr learned my addr 172.21.15.19:0/1496775600 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:02.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.036+0000 7fe3f17fa640 10 client.?.objecter ms_handle_connect 0x7fe3f41430c0 2024-03-25T19:34:02.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.036+0000 7fe3f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:02.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.037+0000 7fe3f17fa640 10 client.?.objecter ms_handle_connect 0x7fe3f40a98f0 2024-03-25T19:34:02.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.037+0000 7fe3f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:02.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.051+0000 7fe3f17fa640 10 client.?.objecter ms_handle_connect 0x7fe3f413f710 2024-03-25T19:34:02.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.051+0000 7fe3f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:02.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.058+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764563666 0 0) 0x7fe3f4132a30 con 0x7fe3f413f710 2024-03-25T19:34:02.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.059+0000 7fe3f17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:02.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.059+0000 7fe3f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:02.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.059+0000 7fe3f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:02.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.059+0000 7fe3f17fa640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:34:02.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.059+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3d8004eb0 con 0x7fe3f413f710 2024-03-25T19:34:02.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.059+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155068921 0 0) 0x7fe3f4138c80 con 0x7fe3f41430c0 2024-03-25T19:34:02.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.059+0000 7fe3f17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:02.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.059+0000 7fe3f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:02.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.059+0000 7fe3f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:02.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.059+0000 7fe3f17fa640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:34:02.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.059+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f4132a30 con 0x7fe3f41430c0 2024-03-25T19:34:02.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.061+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306196558 0 0) 0x7fe3f4138f60 con 0x7fe3f40a98f0 2024-03-25T19:34:02.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.061+0000 7fe3f17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:02.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.061+0000 7fe3f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:02.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.061+0000 7fe3f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:02.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.061+0000 7fe3f17fa640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:34:02.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.061+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f4138c80 con 0x7fe3f40a98f0 2024-03-25T19:34:02.075 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.075+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3327938196 0 0) 0x7fe3f4132a30 con 0x7fe3f41430c0 2024-03-25T19:34:02.075 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.075+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3f4138f60 con 0x7fe3f41430c0 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.075+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe3ec001bd0 con 0x7fe3f41430c0 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.075+0000 7fe3f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.075+0000 7fe3f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.075+0000 7fe3f17fa640 10 monclient(hunting): dump: 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:02.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.075+0000 7fe3f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:02.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.084+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3845659003 0 0) 0x7fe3f4138c80 con 0x7fe3f40a98f0 2024-03-25T19:34:02.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.084+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3f4132a30 con 0x7fe3f40a98f0 2024-03-25T19:34:02.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.084+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe3e8001b00 con 0x7fe3f40a98f0 2024-03-25T19:34:02.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.084+0000 7fe3f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:02.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.084+0000 7fe3f17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:02.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.084+0000 7fe3f17fa640 10 monclient(hunting): dump: 2024-03-25T19:34:02.084 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:02.084 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:02.084 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:02.085 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:02.085 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:02.085 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:02.085 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:02.085 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:02.085 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:02.085 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:02.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4014523362 0 0) 0x7fe3f4138f60 con 0x7fe3f41430c0 2024-03-25T19:34:02.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 >> v1:172.21.15.19:6790/0 conn(0x7fe3f40a98f0 legacy=0x7fe3f4137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 >> v1:172.21.15.19:6789/0 conn(0x7fe3f413f710 legacy=0x7fe3f4131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3f17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:34:02.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3f17fa640 1 monclient: found mon.b 2024-03-25T19:34:02.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:02.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3f17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:02.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f4139240 con 0x7fe3f41430c0 2024-03-25T19:34:02.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3f17fa640 10 monclient: _finish_auth 0 2024-03-25T19:34:02.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3f17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:34:02.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3fae5c640 5 monclient: authenticate success, global_id 4450 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3fae5c640 10 monclient: _renew_subs 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3fae5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3fae5c640 1 -- 172.21.15.19:0/1496775600 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f4136e50 con 0x7fe3f41430c0 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3fae5c640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3fae5c640 10 monclient: _renew_subs 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3fae5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3fae5c640 1 -- 172.21.15.19:0/1496775600 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f41389f0 con 0x7fe3f41430c0 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.085+0000 7fe3fae5c640 1 librados: init done 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.086+0000 7fe3fae5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.086+0000 7fe3fae5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.086+0000 7fe3fae5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:02.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.086+0000 7fe3fae5c640 1 -- 172.21.15.19:0/1496775600 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f40627b0 con 0x7fe3f41430c0 2024-03-25T19:34:02.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.396+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3ec003990 con 0x7fe3f41430c0 2024-03-25T19:34:02.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.396+0000 7fe3f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:02.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.396+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe3ec001d30 con 0x7fe3f41430c0 2024-03-25T19:34:02.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.396+0000 7fe3f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:02.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.396+0000 7fe3f17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:02.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.396+0000 7fe3f2ffd640 4 set_mon_vals no callback set 2024-03-25T19:34:02.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.396+0000 7fe3f17fa640 10 monclient: dump: 2024-03-25T19:34:02.397 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:02.397 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:02.397 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:02.397 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:02.398 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:02.398 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:02.398 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:02.398 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:02.398 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:02.398 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:02.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.397+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fe3ec003de0 con 0x7fe3f41430c0 2024-03-25T19:34:02.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.399+0000 7fe3f8bf8640 10 monclient: get_auth_request con 0x7fe3d806df80 auth_method 0 2024-03-25T19:34:02.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.400+0000 7fe3f17fa640 10 client.4450.objecter ms_handle_connect 0x7fe3d806df80 2024-03-25T19:34:02.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.403+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fe3ec0549c0 con 0x7fe3f41430c0 2024-03-25T19:34:02.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.403+0000 7fe3f17fa640 10 client.4450.objecter ms_dispatch 0x7fe3f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:02.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.403+0000 7fe3f17fa640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:02.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.403+0000 7fe3f17fa640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:02.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.403+0000 7fe3f17fa640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-25T19:34:02.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.404+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3ec0504d0 con 0x7fe3f41430c0 2024-03-25T19:34:02.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.404+0000 7fe3f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:02.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.404+0000 7fe3f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:02.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.671+0000 7fe3fae5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:02.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.671+0000 7fe3fae5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:02.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.671+0000 7fe3fae5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:02.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.671+0000 7fe3fae5c640 1 -- 172.21.15.19:0/1496775600 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3f413dd60 con 0x7fe3f41430c0 2024-03-25T19:34:02.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.707+0000 7fe3f17fa640 1 -- 172.21.15.19:0/1496775600 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fe3ec052400 con 0x7fe3f41430c0 2024-03-25T19:34:02.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.708+0000 7fe3f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:02.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.708+0000 7fe3f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:02.708 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:34:02.708 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.710+0000 7fe3fae5c640 10 librados: watch_flush enter 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.710+0000 7fe3fae5c640 10 librados: watch_flush exit 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.710+0000 7fe3fae5c640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.710+0000 7fe3fae5c640 10 client.4450.objecter successfully canceled tick 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.710+0000 7fe3fae5c640 1 -- 172.21.15.19:0/1496775600 >> v1:172.21.15.188:6800/2207506229 conn(0x7fe3d806df80 legacy=0x7fe3d8070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.710+0000 7fe3fae5c640 10 monclient: shutdown 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.710+0000 7fe3fae5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.710+0000 7fe3fae5c640 1 -- 172.21.15.19:0/1496775600 >> v1:172.21.15.188:6789/0 conn(0x7fe3f41430c0 legacy=0x7fe3f4132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.710+0000 7fe3fae5c640 1 -- 172.21.15.19:0/1496775600 shutdown_connections 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.710+0000 7fe3fae5c640 1 -- 172.21.15.19:0/1496775600 >> 172.21.15.19:0/1496775600 conn(0x7fe3f40b7ea0 msgr2=0x7fe3f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.711+0000 7fe3fae5c640 1 -- 172.21.15.19:0/1496775600 shutdown_connections 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.711+0000 7fe3fae5c640 1 -- 172.21.15.19:0/1496775600 wait complete. 2024-03-25T19:34:02.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:02.711+0000 7fe3fae5c640 1 librados: shutdown 2024-03-25T19:34:02.721 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:09.527 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:09.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.642+0000 7f3e99d0e640 10 monclient: get_monmap_and_config 2024-03-25T19:34:09.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.642+0000 7f3e99d0e640 10 monclient: build_initial_monmap 2024-03-25T19:34:09.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.642+0000 7f3e99d0e640 10 monclient: monmap: 2024-03-25T19:34:09.642 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:09.643 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:09.643 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:34:09.643633+0000 2024-03-25T19:34:09.643 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:34:09.643633+0000 2024-03-25T19:34:09.643 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:09.643 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:09.643 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:09.643 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:34:09.643 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:34:09.643 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:09.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.643+0000 7f3e99d0e640 1 Processor -- start 2024-03-25T19:34:09.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.643+0000 7f3e99d0e640 1 -- start start 2024-03-25T19:34:09.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.643+0000 7f3e99d0e640 10 monclient: init 2024-03-25T19:34:09.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.643+0000 7f3e99d0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:09.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.643+0000 7f3e99d0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:09.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.643+0000 7f3e99d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f3e940b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:09.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.643+0000 7f3e99d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f3e940b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:09.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.643+0000 7f3e99d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f3e9414be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:09.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.643+0000 7f3e99d0e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e940af940 con 0x7f3e940b1730 2024-03-25T19:34:09.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.643+0000 7f3e99d0e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e940b0a70 con 0x7f3e940b4d60 2024-03-25T19:34:09.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.643+0000 7f3e99d0e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e940b0d50 con 0x7f3e9414be90 2024-03-25T19:34:09.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.643+0000 7f3e99d0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:09.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.644+0000 7f3e99d0e640 10 monclient(hunting): authenticate will time out at 6302.913919s 2024-03-25T19:34:09.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.656+0000 7f3e92ffd640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f3e940b1730 0x7f3e940b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:39466/0 (socket says 172.21.15.19:39466) 2024-03-25T19:34:09.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.656+0000 7f3e92ffd640 1 -- 172.21.15.19:0/1352947033 learned_addr learned my addr 172.21.15.19:0/1352947033 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:09.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.656+0000 7f3e937fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f3e940b4d60 0x7f3e9413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:41092/0 (socket says 172.21.15.19:41092) 2024-03-25T19:34:09.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.669+0000 7f3e927fc640 1 -- 172.21.15.19:0/1352947033 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552401763 0 0) 0x7f3e940af940 con 0x7f3e940b1730 2024-03-25T19:34:09.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.669+0000 7f3e927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:09.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.669+0000 7f3e927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:09.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.669+0000 7f3e927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:09.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.669+0000 7f3e927fc640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:34:09.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.669+0000 7f3e927fc640 1 -- 172.21.15.19:0/1352947033 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e7c003520 con 0x7f3e940b1730 2024-03-25T19:34:09.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.670+0000 7f3e927fc640 1 -- 172.21.15.19:0/1352947033 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825447088 0 0) 0x7f3e940b0a70 con 0x7f3e940b4d60 2024-03-25T19:34:09.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.670+0000 7f3e927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:09.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.670+0000 7f3e927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:09.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.670+0000 7f3e927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:09.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.670+0000 7f3e927fc640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:34:09.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.670+0000 7f3e927fc640 1 -- 172.21.15.19:0/1352947033 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e940af940 con 0x7f3e940b4d60 2024-03-25T19:34:09.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 1 -- 172.21.15.19:0/1352947033 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3840258760 0 0) 0x7f3e7c003520 con 0x7f3e940b1730 2024-03-25T19:34:09.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 1 -- 172.21.15.19:0/1352947033 >> v1:172.21.15.19:6790/0 conn(0x7f3e9414be90 legacy=0x7f3e9414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 1 -- 172.21.15.19:0/1352947033 >> v1:172.21.15.188:6789/0 conn(0x7f3e940b4d60 legacy=0x7f3e9413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 1 monclient: found mon.noname-a 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 1 -- 172.21.15.19:0/1352947033 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e940b1030 con 0x7f3e940b1730 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 1 -- 172.21.15.19:0/1352947033 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3e88001690 con 0x7f3e940b1730 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e927fc640 10 monclient: dump: 2024-03-25T19:34:09.682 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:09.683 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:09.683 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:09.683 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:09.683 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:09.683 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:09.683 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:09.683 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:09.683 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:09.683 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:09.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e99d0e640 5 monclient: authenticate success, global_id 4458 2024-03-25T19:34:09.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.681+0000 7f3e99d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:09.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.688+0000 7f3e927fc640 1 -- 172.21.15.19:0/1352947033 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e880017f0 con 0x7f3e940b1730 2024-03-25T19:34:09.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.688+0000 7f3e927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:09.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.688+0000 7f3e927fc640 1 -- 172.21.15.19:0/1352947033 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3e88001c00 con 0x7f3e940b1730 2024-03-25T19:34:09.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.688+0000 7f3e927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.688+0000 7f3e927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.688+0000 7f3e927fc640 10 monclient: dump: 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.688+0000 7f3e99d0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:09.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.688+0000 7f3e99d0e640 4 set_mon_vals no callback set 2024-03-25T19:34:09.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.689+0000 7f3e99d0e640 10 monclient: shutdown 2024-03-25T19:34:09.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.689+0000 7f3e99d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:09.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.689+0000 7f3e99d0e640 1 -- 172.21.15.19:0/1352947033 >> v1:172.21.15.19:6789/0 conn(0x7f3e940b1730 legacy=0x7f3e940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.689+0000 7f3e99d0e640 1 -- 172.21.15.19:0/1352947033 shutdown_connections 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.689+0000 7f3e99d0e640 1 -- 172.21.15.19:0/1352947033 >> 172.21.15.19:0/1352947033 conn(0x7f3e940617a0 msgr2=0x7f3e94061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.689+0000 7f3e99d0e640 1 -- 172.21.15.19:0/1352947033 shutdown_connections 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.689+0000 7f3e99d0e640 1 -- 172.21.15.19:0/1352947033 wait complete. 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.690+0000 7f3e99d0e640 10 monclient: build_initial_monmap 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.690+0000 7f3e99d0e640 10 monclient: monmap: 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:34:09.692 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.690+0000 7f3e99d0e640 1 librados: starting msgr at 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.690+0000 7f3e99d0e640 1 librados: starting objecter 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.690+0000 7f3e99d0e640 1 Processor -- start 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.690+0000 7f3e99d0e640 1 -- start start 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.690+0000 7f3e99d0e640 1 librados: setting wanted keys 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.690+0000 7f3e99d0e640 1 librados: calling monclient init 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.690+0000 7f3e99d0e640 10 monclient: init 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.691+0000 7f3e99d0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.691+0000 7f3e99d0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.691+0000 7f3e99d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f3e940b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.691+0000 7f3e99d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f3e940b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:09.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.691+0000 7f3e99d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f3e9414be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:09.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.691+0000 7f3e99d0e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e941daec0 con 0x7f3e940b1730 2024-03-25T19:34:09.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.691+0000 7f3e99d0e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e941dbd40 con 0x7f3e940b4d60 2024-03-25T19:34:09.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.691+0000 7f3e99d0e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9413a510 con 0x7f3e9414be90 2024-03-25T19:34:09.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.691+0000 7f3e99d0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:09.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.691+0000 7f3e99d0e640 10 monclient(hunting): authenticate will time out at 6302.961434s 2024-03-25T19:34:09.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.700+0000 7f3e937fe640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f3e940b1730 0x7f3e941db630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:39504/0 (socket says 172.21.15.19:39504) 2024-03-25T19:34:09.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.700+0000 7f3e937fe640 1 -- 172.21.15.19:0/486890455 learned_addr learned my addr 172.21.15.19:0/486890455 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:09.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.700+0000 7f3e7bfff640 10 client.?.objecter ms_handle_connect 0x7f3e940b1730 2024-03-25T19:34:09.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.700+0000 7f3e7bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:09.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.706+0000 7f3e7bfff640 10 client.?.objecter ms_handle_connect 0x7f3e9414be90 2024-03-25T19:34:09.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.706+0000 7f3e7bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:09.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.708+0000 7f3e7bfff640 10 client.?.objecter ms_handle_connect 0x7f3e940b4d60 2024-03-25T19:34:09.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.708+0000 7f3e7bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:09.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.719+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148350327 0 0) 0x7f3e941dbd40 con 0x7f3e940b4d60 2024-03-25T19:34:09.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.719+0000 7f3e7bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:09.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.719+0000 7f3e7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:09.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.719+0000 7f3e7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:09.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.719+0000 7f3e7bfff640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:34:09.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.719+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e80004eb0 con 0x7f3e940b4d60 2024-03-25T19:34:09.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.719+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797837660 0 0) 0x7f3e9413a510 con 0x7f3e9414be90 2024-03-25T19:34:09.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.719+0000 7f3e7bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:09.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.719+0000 7f3e7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:09.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.719+0000 7f3e7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:09.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.719+0000 7f3e7bfff640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:34:09.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.719+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e941dbd40 con 0x7f3e9414be90 2024-03-25T19:34:09.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.723+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276955030 0 0) 0x7f3e941daec0 con 0x7f3e940b1730 2024-03-25T19:34:09.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.723+0000 7f3e7bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:09.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.723+0000 7f3e7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:09.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.723+0000 7f3e7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:09.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.723+0000 7f3e7bfff640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:34:09.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.723+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e9413a510 con 0x7f3e940b1730 2024-03-25T19:34:09.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.739+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1039884543 0 0) 0x7f3e80004eb0 con 0x7f3e940b4d60 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.739+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e941daec0 con 0x7f3e940b4d60 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.739+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3e88001b00 con 0x7f3e940b4d60 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.739+0000 7f3e7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.739+0000 7f3e7bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.739+0000 7f3e7bfff640 10 monclient(hunting): dump: 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:09.740 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:09.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.739+0000 7f3e7bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.742+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3279447985 0 0) 0x7f3e941dbd40 con 0x7f3e9414be90 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.742+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e80004eb0 con 0x7f3e9414be90 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.742+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3e8c001b10 con 0x7f3e9414be90 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.742+0000 7f3e7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.742+0000 7f3e7bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.742+0000 7f3e7bfff640 10 monclient(hunting): dump: 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:09.742 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:09.743 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:09.743 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:09.743 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:09.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1862351853 0 0) 0x7f3e941daec0 con 0x7f3e940b4d60 2024-03-25T19:34:09.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 >> v1:172.21.15.19:6790/0 conn(0x7f3e9414be90 legacy=0x7f3e941e3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:09.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 >> v1:172.21.15.19:6789/0 conn(0x7f3e940b1730 legacy=0x7f3e941db630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:09.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e7bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:09.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e7bfff640 1 monclient: found mon.b 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e7bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9413a7f0 con 0x7f3e940b4d60 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e7bfff640 10 monclient: _finish_auth 0 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e7bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e99d0e640 5 monclient: authenticate success, global_id 4456 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e99d0e640 10 monclient: _renew_subs 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e99d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e99d0e640 1 -- 172.21.15.19:0/486890455 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e941db110 con 0x7f3e940b4d60 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e99d0e640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e99d0e640 10 monclient: _renew_subs 2024-03-25T19:34:09.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e99d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:09.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e99d0e640 1 -- 172.21.15.19:0/486890455 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9413c090 con 0x7f3e940b4d60 2024-03-25T19:34:09.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.757+0000 7f3e99d0e640 1 librados: init done 2024-03-25T19:34:09.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.758+0000 7f3e99d0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:09.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.758+0000 7f3e99d0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:09.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.758+0000 7f3e99d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:09.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.758+0000 7f3e99d0e640 1 -- 172.21.15.19:0/486890455 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e940627b0 con 0x7f3e940b4d60 2024-03-25T19:34:09.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.767+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e88001220 con 0x7f3e940b4d60 2024-03-25T19:34:09.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.767+0000 7f3e7bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:09.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:09.767+0000 7f3e91ffb640 4 set_mon_vals no callback set 2024-03-25T19:34:10.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.030+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3e88004c30 con 0x7f3e940b4d60 2024-03-25T19:34:10.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.030+0000 7f3e7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:10.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.030+0000 7f3e7bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:10.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.030+0000 7f3e7bfff640 10 monclient: dump: 2024-03-25T19:34:10.031 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:10.031 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:10.031 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:10.031 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:10.031 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:10.031 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:10.031 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:10.032 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:10.032 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:10.032 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:10.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.031+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f3e8801ac00 con 0x7f3e940b4d60 2024-03-25T19:34:10.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.032+0000 7f3e937fe640 10 monclient: get_auth_request con 0x7f3e8006df60 auth_method 0 2024-03-25T19:34:10.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.032+0000 7f3e7bfff640 10 client.4456.objecter ms_handle_connect 0x7f3e8006df60 2024-03-25T19:34:10.034 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.034+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f3e880548f0 con 0x7f3e940b4d60 2024-03-25T19:34:10.034 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.034+0000 7f3e7bfff640 10 client.4456.objecter ms_dispatch 0x7f3e94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:10.034 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.034+0000 7f3e7bfff640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:10.034 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.034+0000 7f3e7bfff640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:10.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.034+0000 7f3e7bfff640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-25T19:34:10.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.037+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e880b18b0 con 0x7f3e940b4d60 2024-03-25T19:34:10.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.037+0000 7f3e7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:10.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.037+0000 7f3e7bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:10.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.288+0000 7f3e99d0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:10.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.288+0000 7f3e99d0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:10.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.288+0000 7f3e99d0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:10.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.288+0000 7f3e99d0e640 1 -- 172.21.15.19:0/486890455 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e94069f40 con 0x7f3e940b4d60 2024-03-25T19:34:10.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.328+0000 7f3e7bfff640 1 -- 172.21.15.19:0/486890455 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f3e88050230 con 0x7f3e940b4d60 2024-03-25T19:34:10.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.328+0000 7f3e7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:10.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.328+0000 7f3e7bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:10.328 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:34:10.329 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:10.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.330+0000 7f3e99d0e640 10 librados: watch_flush enter 2024-03-25T19:34:10.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.330+0000 7f3e99d0e640 10 librados: watch_flush exit 2024-03-25T19:34:10.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.330+0000 7f3e99d0e640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-25T19:34:10.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.330+0000 7f3e99d0e640 10 client.4456.objecter successfully canceled tick 2024-03-25T19:34:10.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.330+0000 7f3e99d0e640 1 -- 172.21.15.19:0/486890455 >> v1:172.21.15.188:6800/2207506229 conn(0x7f3e8006df60 legacy=0x7f3e80070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:10.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.330+0000 7f3e99d0e640 10 monclient: shutdown 2024-03-25T19:34:10.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.330+0000 7f3e99d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:10.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.330+0000 7f3e99d0e640 1 -- 172.21.15.19:0/486890455 >> v1:172.21.15.188:6789/0 conn(0x7f3e940b4d60 legacy=0x7f3e9413b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:10.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.331+0000 7f3e99d0e640 1 -- 172.21.15.19:0/486890455 shutdown_connections 2024-03-25T19:34:10.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.331+0000 7f3e99d0e640 1 -- 172.21.15.19:0/486890455 >> 172.21.15.19:0/486890455 conn(0x7f3e940617a0 msgr2=0x7f3e940b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:10.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.331+0000 7f3e99d0e640 1 -- 172.21.15.19:0/486890455 shutdown_connections 2024-03-25T19:34:10.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.331+0000 7f3e99d0e640 1 -- 172.21.15.19:0/486890455 wait complete. 2024-03-25T19:34:10.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:10.331+0000 7f3e99d0e640 1 librados: shutdown 2024-03-25T19:34:10.344 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:12.573 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:35.302 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.584 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:35.314 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.587 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:35.316 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.608 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:35.337 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.630 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:05:35.333 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.638 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:05:35.341 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.651 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:35.380 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.669 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:05:35.372 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.687 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:35.416 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:17.151 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:17.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.266+0000 7f58cfa0d640 10 monclient: get_monmap_and_config 2024-03-25T19:34:17.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.266+0000 7f58cfa0d640 10 monclient: build_initial_monmap 2024-03-25T19:34:17.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.267+0000 7f58cfa0d640 10 monclient: monmap: 2024-03-25T19:34:17.267 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:17.267 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:17.267 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:34:17.268221+0000 2024-03-25T19:34:17.267 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:34:17.268221+0000 2024-03-25T19:34:17.267 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:17.267 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:17.267 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:17.267 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:34:17.267 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:34:17.267 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:17.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.267+0000 7f58cfa0d640 1 Processor -- start 2024-03-25T19:34:17.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.267+0000 7f58cfa0d640 1 -- start start 2024-03-25T19:34:17.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.267+0000 7f58cfa0d640 10 monclient: init 2024-03-25T19:34:17.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.268+0000 7f58cfa0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:17.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.268+0000 7f58cfa0d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:17.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.268+0000 7f58cfa0d640 10 monclient(hunting): picked mon.noname-c con 0x7f58c80a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:17.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.268+0000 7f58cfa0d640 10 monclient(hunting): picked mon.noname-a con 0x7f58c813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:17.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.268+0000 7f58cfa0d640 10 monclient(hunting): picked mon.noname-b con 0x7f58c81430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:17.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.268+0000 7f58cfa0d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c8142520 con 0x7f58c813f710 2024-03-25T19:34:17.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.268+0000 7f58cfa0d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c80b28a0 con 0x7f58c80a98f0 2024-03-25T19:34:17.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.268+0000 7f58cfa0d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c80b2a30 con 0x7f58c81430c0 2024-03-25T19:34:17.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.268+0000 7f58cfa0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:17.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.268+0000 7f58cfa0d640 10 monclient(hunting): authenticate will time out at 6310.538569s 2024-03-25T19:34:17.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.274+0000 7f58cd7a9640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f58c80a98f0 0x7f58c80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:45708/0 (socket says 172.21.15.19:45708) 2024-03-25T19:34:17.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.274+0000 7f58cd7a9640 1 -- 172.21.15.19:0/1676369859 learned_addr learned my addr 172.21.15.19:0/1676369859 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:17.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.283+0000 7f58bffff640 1 -- 172.21.15.19:0/1676369859 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304123485 0 0) 0x7f58c80b28a0 con 0x7f58c80a98f0 2024-03-25T19:34:17.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.283+0000 7f58bffff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:17.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.283+0000 7f58bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:17.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.283+0000 7f58bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:17.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.283+0000 7f58bffff640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:34:17.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.283+0000 7f58bffff640 1 -- 172.21.15.19:0/1676369859 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58b4003520 con 0x7f58c80a98f0 2024-03-25T19:34:17.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.290+0000 7f58bffff640 1 -- 172.21.15.19:0/1676369859 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788996032 0 0) 0x7f58c8142520 con 0x7f58c813f710 2024-03-25T19:34:17.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.290+0000 7f58bffff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:17.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.290+0000 7f58bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:17.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.290+0000 7f58bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:17.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.290+0000 7f58bffff640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:34:17.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.290+0000 7f58bffff640 1 -- 172.21.15.19:0/1676369859 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c80b28a0 con 0x7f58c813f710 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.303+0000 7f58bffff640 1 -- 172.21.15.19:0/1676369859 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2526686723 0 0) 0x7f58c80b28a0 con 0x7f58c813f710 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.303+0000 7f58bffff640 1 -- 172.21.15.19:0/1676369859 >> v1:172.21.15.19:6790/0 conn(0x7f58c81430c0 legacy=0x7f58c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.303+0000 7f58bffff640 1 -- 172.21.15.19:0/1676369859 >> v1:172.21.15.188:6789/0 conn(0x7f58c80a98f0 legacy=0x7f58c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.303+0000 7f58bffff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.303+0000 7f58bffff640 1 monclient: found mon.noname-a 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.303+0000 7f58bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.303+0000 7f58bffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.303+0000 7f58bffff640 1 -- 172.21.15.19:0/1676369859 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c80b44c0 con 0x7f58c813f710 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.303+0000 7f58bffff640 10 monclient: _finish_auth 0 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.303+0000 7f58bffff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.303+0000 7f58bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.304+0000 7f58cfa0d640 5 monclient: authenticate success, global_id 4464 2024-03-25T19:34:17.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.304+0000 7f58cfa0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:17.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.305+0000 7f58bffff640 1 -- 172.21.15.19:0/1676369859 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f58b8001630 con 0x7f58c813f710 2024-03-25T19:34:17.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.305+0000 7f58bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:17.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.305+0000 7f58bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:17.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.305+0000 7f58bffff640 10 monclient: dump: 2024-03-25T19:34:17.306 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:17.306 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:17.306 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:17.306 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:17.306 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:17.306 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:17.306 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:17.306 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:17.306 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:17.306 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:17.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.305+0000 7f58cfa0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:17.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.318+0000 7f58bffff640 1 -- 172.21.15.19:0/1676369859 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58b80018c0 con 0x7f58c813f710 2024-03-25T19:34:17.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.318+0000 7f58bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:17.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.318+0000 7f58bffff640 1 -- 172.21.15.19:0/1676369859 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f58b8001bd0 con 0x7f58c813f710 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.318+0000 7f58cfa0d640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.318+0000 7f58cfa0d640 4 set_mon_vals no callback set 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.318+0000 7f58bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.318+0000 7f58bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.318+0000 7f58bffff640 10 monclient: dump: 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:17.319 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:17.320 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:17.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.319+0000 7f58cfa0d640 10 monclient: shutdown 2024-03-25T19:34:17.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.319+0000 7f58cfa0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:17.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.319+0000 7f58cfa0d640 1 -- 172.21.15.19:0/1676369859 >> v1:172.21.15.19:6789/0 conn(0x7f58c813f710 legacy=0x7f58c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:17.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.319+0000 7f58cfa0d640 1 -- 172.21.15.19:0/1676369859 shutdown_connections 2024-03-25T19:34:17.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.319+0000 7f58cfa0d640 1 -- 172.21.15.19:0/1676369859 >> 172.21.15.19:0/1676369859 conn(0x7f58c80b7ea0 msgr2=0x7f58c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:17.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.319+0000 7f58cfa0d640 1 -- 172.21.15.19:0/1676369859 shutdown_connections 2024-03-25T19:34:17.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.319+0000 7f58cfa0d640 1 -- 172.21.15.19:0/1676369859 wait complete. 2024-03-25T19:34:17.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.320+0000 7f58cfa0d640 10 monclient: build_initial_monmap 2024-03-25T19:34:17.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.320+0000 7f58cfa0d640 10 monclient: monmap: 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.321+0000 7f58cfa0d640 1 librados: starting msgr at 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.321+0000 7f58cfa0d640 1 librados: starting objecter 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.321+0000 7f58cfa0d640 1 Processor -- start 2024-03-25T19:34:17.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.321+0000 7f58cfa0d640 1 -- start start 2024-03-25T19:34:17.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.321+0000 7f58cfa0d640 1 librados: setting wanted keys 2024-03-25T19:34:17.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.321+0000 7f58cfa0d640 1 librados: calling monclient init 2024-03-25T19:34:17.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.321+0000 7f58cfa0d640 10 monclient: init 2024-03-25T19:34:17.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.322+0000 7f58cfa0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:17.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.322+0000 7f58cfa0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:17.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.322+0000 7f58cfa0d640 10 monclient(hunting): picked mon.noname-c con 0x7f58c80a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:17.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.322+0000 7f58cfa0d640 10 monclient(hunting): picked mon.noname-a con 0x7f58c813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:17.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.322+0000 7f58cfa0d640 10 monclient(hunting): picked mon.noname-b con 0x7f58c81430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:17.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.322+0000 7f58cfa0d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c814d160 con 0x7f58c813f710 2024-03-25T19:34:17.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.322+0000 7f58cfa0d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c80b7930 con 0x7f58c81430c0 2024-03-25T19:34:17.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.322+0000 7f58cfa0d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c814c000 con 0x7f58c80a98f0 2024-03-25T19:34:17.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.322+0000 7f58cfa0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:17.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.322+0000 7f58cfa0d640 10 monclient(hunting): authenticate will time out at 6310.592414s 2024-03-25T19:34:17.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.332+0000 7f58cd7a9640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f58c80a98f0 0x7f58c8134f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:57384/0 (socket says 172.21.15.19:57384) 2024-03-25T19:34:17.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.332+0000 7f58cd7a9640 1 -- 172.21.15.19:0/4115709921 learned_addr learned my addr 172.21.15.19:0/4115709921 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:17.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.333+0000 7f58bdffb640 10 client.?.objecter ms_handle_connect 0x7f58c80a98f0 2024-03-25T19:34:17.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.333+0000 7f58bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:17.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.338+0000 7f58bdffb640 10 client.?.objecter ms_handle_connect 0x7f58c813f710 2024-03-25T19:34:17.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.338+0000 7f58bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:17.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.339+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607502729 0 0) 0x7f58c814c000 con 0x7f58c80a98f0 2024-03-25T19:34:17.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.339+0000 7f58bdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:17.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.339+0000 7f58bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:17.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.339+0000 7f58bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:17.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.339+0000 7f58bdffb640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:34:17.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.339+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ac004eb0 con 0x7f58c80a98f0 2024-03-25T19:34:17.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.339+0000 7f58bdffb640 10 client.?.objecter ms_handle_connect 0x7f58c81430c0 2024-03-25T19:34:17.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.339+0000 7f58bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:17.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.347+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1730669807 0 0) 0x7f58ac004eb0 con 0x7f58c80a98f0 2024-03-25T19:34:17.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.347+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58c814c000 con 0x7f58c80a98f0 2024-03-25T19:34:17.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.347+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f58c4001950 con 0x7f58c80a98f0 2024-03-25T19:34:17.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.347+0000 7f58bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:17.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.347+0000 7f58bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:17.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.347+0000 7f58bdffb640 10 monclient(hunting): dump: 2024-03-25T19:34:17.347 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:17.347 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:17.348 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:17.348 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:17.348 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:17.348 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:17.348 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:17.348 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:17.348 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:17.348 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:17.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.347+0000 7f58bdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:17.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.350+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840935800 0 0) 0x7f58c80b7930 con 0x7f58c81430c0 2024-03-25T19:34:17.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.350+0000 7f58bdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:17.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.350+0000 7f58bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:17.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.350+0000 7f58bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:17.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.350+0000 7f58bdffb640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:34:17.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.350+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ac004eb0 con 0x7f58c81430c0 2024-03-25T19:34:17.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.360+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832642429 0 0) 0x7f58c814d160 con 0x7f58c813f710 2024-03-25T19:34:17.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.360+0000 7f58bdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:17.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.360+0000 7f58bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:17.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.360+0000 7f58bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:17.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.360+0000 7f58bdffb640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:34:17.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.360+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c80b7930 con 0x7f58c813f710 2024-03-25T19:34:17.364 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4041058654 0 0) 0x7f58c814c000 con 0x7f58c80a98f0 2024-03-25T19:34:17.364 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 >> v1:172.21.15.188:6789/0 conn(0x7f58c81430c0 legacy=0x7f58c81dbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:17.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 >> v1:172.21.15.19:6789/0 conn(0x7f58c813f710 legacy=0x7f58c81db720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:17.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58bdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:34:17.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58bdffb640 1 monclient: found mon.c 2024-03-25T19:34:17.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:17.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58bdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:17.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c814c190 con 0x7f58c80a98f0 2024-03-25T19:34:17.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58bdffb640 10 monclient: _finish_auth 0 2024-03-25T19:34:17.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58bdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:34:17.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:17.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58cfa0d640 5 monclient: authenticate success, global_id 4442 2024-03-25T19:34:17.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58cfa0d640 10 monclient: _renew_subs 2024-03-25T19:34:17.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58cfa0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:17.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58cfa0d640 1 -- 172.21.15.19:0/4115709921 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c80b7380 con 0x7f58c80a98f0 2024-03-25T19:34:17.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58cfa0d640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:17.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58cfa0d640 10 monclient: _renew_subs 2024-03-25T19:34:17.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58cfa0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:17.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58cfa0d640 1 -- 172.21.15.19:0/4115709921 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c813abf0 con 0x7f58c80a98f0 2024-03-25T19:34:17.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.364+0000 7f58cfa0d640 1 librados: init done 2024-03-25T19:34:17.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.365+0000 7f58cfa0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:17.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.365+0000 7f58cfa0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:17.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.365+0000 7f58cfa0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:17.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.365+0000 7f58cfa0d640 1 -- 172.21.15.19:0/4115709921 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c80b3ef0 con 0x7f58c80a98f0 2024-03-25T19:34:17.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.675+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58c40047e0 con 0x7f58c80a98f0 2024-03-25T19:34:17.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.675+0000 7f58bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:17.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.675+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f58c4004b70 con 0x7f58c80a98f0 2024-03-25T19:34:17.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.675+0000 7f58bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:17.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.675+0000 7f58bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:17.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.675+0000 7f58bdffb640 10 monclient: dump: 2024-03-25T19:34:17.676 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:17.676 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:17.676 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:17.676 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:17.676 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:17.676 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:17.676 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:17.677 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:17.677 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:17.677 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:17.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.675+0000 7f58bf7fe640 4 set_mon_vals no callback set 2024-03-25T19:34:17.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.676+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f58c4004d90 con 0x7f58c80a98f0 2024-03-25T19:34:17.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.677+0000 7f58ccfa8640 10 monclient: get_auth_request con 0x7f58ac06de10 auth_method 0 2024-03-25T19:34:17.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.677+0000 7f58bdffb640 10 client.4442.objecter ms_handle_connect 0x7f58ac06de10 2024-03-25T19:34:17.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.684+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f58c4054800 con 0x7f58c80a98f0 2024-03-25T19:34:17.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.684+0000 7f58bdffb640 10 client.4442.objecter ms_dispatch 0x7f58c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:17.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.684+0000 7f58bdffb640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:17.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.684+0000 7f58bdffb640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:17.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.684+0000 7f58bdffb640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-25T19:34:17.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.684+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58c4050310 con 0x7f58c80a98f0 2024-03-25T19:34:17.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.684+0000 7f58bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:17.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.684+0000 7f58bdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:17.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.936+0000 7f58cfa0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:17.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.936+0000 7f58cfa0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:17.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.936+0000 7f58cfa0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:17.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:17.936+0000 7f58cfa0d640 1 -- 172.21.15.19:0/4115709921 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58c80b7ac0 con 0x7f58c80a98f0 2024-03-25T19:34:18.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.055+0000 7f58bdffb640 1 -- 172.21.15.19:0/4115709921 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f58c4052240 con 0x7f58c80a98f0 2024-03-25T19:34:18.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.055+0000 7f58bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:18.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.055+0000 7f58bdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:18.055 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:34:18.055 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:18.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.057+0000 7f58cfa0d640 10 librados: watch_flush enter 2024-03-25T19:34:18.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.057+0000 7f58cfa0d640 10 librados: watch_flush exit 2024-03-25T19:34:18.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.057+0000 7f58cfa0d640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-25T19:34:18.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.057+0000 7f58cfa0d640 10 client.4442.objecter successfully canceled tick 2024-03-25T19:34:18.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.057+0000 7f58cfa0d640 1 -- 172.21.15.19:0/4115709921 >> v1:172.21.15.188:6800/2207506229 conn(0x7f58ac06de10 legacy=0x7f58ac0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:18.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.057+0000 7f58cfa0d640 10 monclient: shutdown 2024-03-25T19:34:18.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.057+0000 7f58cfa0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:18.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.057+0000 7f58cfa0d640 1 -- 172.21.15.19:0/4115709921 >> v1:172.21.15.19:6790/0 conn(0x7f58c80a98f0 legacy=0x7f58c8134f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:18.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.057+0000 7f58cfa0d640 1 -- 172.21.15.19:0/4115709921 shutdown_connections 2024-03-25T19:34:18.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.057+0000 7f58cfa0d640 1 -- 172.21.15.19:0/4115709921 >> 172.21.15.19:0/4115709921 conn(0x7f58c80b7ea0 msgr2=0x7f58c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:18.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.057+0000 7f58cfa0d640 1 -- 172.21.15.19:0/4115709921 shutdown_connections 2024-03-25T19:34:18.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.057+0000 7f58cfa0d640 1 -- 172.21.15.19:0/4115709921 wait complete. 2024-03-25T19:34:18.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:18.058+0000 7f58cfa0d640 1 librados: shutdown 2024-03-25T19:34:18.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:24.875 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:24.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.973+0000 7ff646e55640 10 monclient: get_monmap_and_config 2024-03-25T19:34:24.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.973+0000 7ff646e55640 10 monclient: build_initial_monmap 2024-03-25T19:34:24.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.973+0000 7ff646e55640 10 monclient: monmap: 2024-03-25T19:34:24.973 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:24.974 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:24.974 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:34:24.974777+0000 2024-03-25T19:34:24.974 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:34:24.974777+0000 2024-03-25T19:34:24.974 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:24.974 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:24.974 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:24.974 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:34:24.974 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:34:24.974 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:24.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.974+0000 7ff646e55640 1 Processor -- start 2024-03-25T19:34:24.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.974+0000 7ff646e55640 1 -- start start 2024-03-25T19:34:24.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.974+0000 7ff646e55640 10 monclient: init 2024-03-25T19:34:24.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.974+0000 7ff646e55640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:24.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.974+0000 7ff646e55640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:24.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.974+0000 7ff646e55640 10 monclient(hunting): picked mon.noname-a con 0x7ff6400b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:24.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.975+0000 7ff646e55640 10 monclient(hunting): picked mon.noname-b con 0x7ff6400b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:24.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.975+0000 7ff646e55640 10 monclient(hunting): picked mon.noname-c con 0x7ff64014be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:24.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.975+0000 7ff646e55640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400af940 con 0x7ff6400b4d60 2024-03-25T19:34:24.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.975+0000 7ff646e55640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400b0a70 con 0x7ff64014be90 2024-03-25T19:34:24.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.975+0000 7ff646e55640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400b0d50 con 0x7ff6400b1730 2024-03-25T19:34:24.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.975+0000 7ff646e55640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:24.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.975+0000 7ff646e55640 10 monclient(hunting): authenticate will time out at 6318.244986s 2024-03-25T19:34:24.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.987+0000 7ff644bf1640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7ff6400b4d60 0x7ff64013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:60576/0 (socket says 172.21.15.19:60576) 2024-03-25T19:34:24.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:24.987+0000 7ff644bf1640 1 -- 172.21.15.19:0/570302752 learned_addr learned my addr 172.21.15.19:0/570302752 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:25.000 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.001+0000 7ff63f7fe640 1 -- 172.21.15.19:0/570302752 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408286346 0 0) 0x7ff6400af940 con 0x7ff6400b4d60 2024-03-25T19:34:25.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.001+0000 7ff63f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:25.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.001+0000 7ff63f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:25.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.001+0000 7ff63f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:25.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.001+0000 7ff63f7fe640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:34:25.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.001+0000 7ff63f7fe640 1 -- 172.21.15.19:0/570302752 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff630003520 con 0x7ff6400b4d60 2024-03-25T19:34:25.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 1 -- 172.21.15.19:0/570302752 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2574496790 0 0) 0x7ff630003520 con 0x7ff6400b4d60 2024-03-25T19:34:25.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 1 -- 172.21.15.19:0/570302752 >> v1:172.21.15.19:6790/0 conn(0x7ff6400b1730 legacy=0x7ff6400b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:34:25.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 1 -- 172.21.15.19:0/570302752 >> v1:172.21.15.188:6789/0 conn(0x7ff64014be90 legacy=0x7ff64014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:25.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 1 monclient: found mon.noname-a 2024-03-25T19:34:25.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:25.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:34:25.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 1 -- 172.21.15.19:0/570302752 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6400b1030 con 0x7ff6400b4d60 2024-03-25T19:34:25.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:25.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:25.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:25.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 1 -- 172.21.15.19:0/570302752 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff634001690 con 0x7ff6400b4d60 2024-03-25T19:34:25.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff646e55640 5 monclient: authenticate success, global_id 4470 2024-03-25T19:34:25.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff646e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:25.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:25.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:25.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff63f7fe640 10 monclient: dump: 2024-03-25T19:34:25.010 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:25.010 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:25.010 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:25.010 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:25.011 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:25.011 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:25.011 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:25.011 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:25.011 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:25.011 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:25.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.008+0000 7ff646e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:25.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.018+0000 7ff63f7fe640 1 -- 172.21.15.19:0/570302752 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff634001860 con 0x7ff6400b4d60 2024-03-25T19:34:25.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.018+0000 7ff63f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:25.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.018+0000 7ff63f7fe640 1 -- 172.21.15.19:0/570302752 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff634001cb0 con 0x7ff6400b4d60 2024-03-25T19:34:25.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.018+0000 7ff646e55640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:25.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.018+0000 7ff646e55640 4 set_mon_vals no callback set 2024-03-25T19:34:25.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.018+0000 7ff63f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:25.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.018+0000 7ff63f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:25.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.018+0000 7ff63f7fe640 10 monclient: dump: 2024-03-25T19:34:25.018 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:25.018 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:25.018 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:25.019 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:25.019 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:25.019 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:25.019 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:25.019 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:25.019 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:25.019 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:25.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.018+0000 7ff646e55640 10 monclient: shutdown 2024-03-25T19:34:25.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.018+0000 7ff646e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.018+0000 7ff646e55640 1 -- 172.21.15.19:0/570302752 >> v1:172.21.15.19:6789/0 conn(0x7ff6400b4d60 legacy=0x7ff64013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.019+0000 7ff646e55640 1 -- 172.21.15.19:0/570302752 shutdown_connections 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.019+0000 7ff646e55640 1 -- 172.21.15.19:0/570302752 >> 172.21.15.19:0/570302752 conn(0x7ff6400617a0 msgr2=0x7ff640061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.019+0000 7ff646e55640 1 -- 172.21.15.19:0/570302752 shutdown_connections 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.019+0000 7ff646e55640 1 -- 172.21.15.19:0/570302752 wait complete. 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.020+0000 7ff646e55640 10 monclient: build_initial_monmap 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.020+0000 7ff646e55640 10 monclient: monmap: 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:34:25.021 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:25.022 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:34:25.022 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:34:25.022 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:25.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.020+0000 7ff646e55640 1 librados: starting msgr at 2024-03-25T19:34:25.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.020+0000 7ff646e55640 1 librados: starting objecter 2024-03-25T19:34:25.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.020+0000 7ff646e55640 1 Processor -- start 2024-03-25T19:34:25.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.020+0000 7ff646e55640 1 -- start start 2024-03-25T19:34:25.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.020+0000 7ff646e55640 1 librados: setting wanted keys 2024-03-25T19:34:25.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.020+0000 7ff646e55640 1 librados: calling monclient init 2024-03-25T19:34:25.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.020+0000 7ff646e55640 10 monclient: init 2024-03-25T19:34:25.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.021+0000 7ff646e55640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:25.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.021+0000 7ff646e55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:25.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.021+0000 7ff646e55640 10 monclient(hunting): picked mon.noname-b con 0x7ff6400b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:25.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.021+0000 7ff646e55640 10 monclient(hunting): picked mon.noname-c con 0x7ff6400b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:25.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.021+0000 7ff646e55640 10 monclient(hunting): picked mon.noname-a con 0x7ff64014be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:25.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.021+0000 7ff646e55640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff640138170 con 0x7ff64014be90 2024-03-25T19:34:25.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.021+0000 7ff646e55640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff640137e40 con 0x7ff6400b1730 2024-03-25T19:34:25.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.021+0000 7ff646e55640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6401324b0 con 0x7ff6400b4d60 2024-03-25T19:34:25.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.021+0000 7ff646e55640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:25.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.021+0000 7ff646e55640 10 monclient(hunting): authenticate will time out at 6318.291432s 2024-03-25T19:34:25.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.031+0000 7ff63ffff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7ff6400b4d60 0x7ff640131da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:54388/0 (socket says 172.21.15.19:54388) 2024-03-25T19:34:25.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.031+0000 7ff63ffff640 1 -- 172.21.15.19:0/2351923287 learned_addr learned my addr 172.21.15.19:0/2351923287 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:25.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.036+0000 7ff63d7fa640 10 client.?.objecter ms_handle_connect 0x7ff6400b1730 2024-03-25T19:34:25.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.036+0000 7ff63d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:25.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.037+0000 7ff63d7fa640 10 client.?.objecter ms_handle_connect 0x7ff6400b4d60 2024-03-25T19:34:25.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.037+0000 7ff63d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:25.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.044+0000 7ff63d7fa640 10 client.?.objecter ms_handle_connect 0x7ff64014be90 2024-03-25T19:34:25.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.044+0000 7ff63d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:25.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.047+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263244222 0 0) 0x7ff640137e40 con 0x7ff6400b1730 2024-03-25T19:34:25.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.047+0000 7ff63d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:25.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.047+0000 7ff63d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:25.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.047+0000 7ff63d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:25.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.047+0000 7ff63d7fa640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:34:25.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.047+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff624004eb0 con 0x7ff6400b1730 2024-03-25T19:34:25.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.057+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209039693 0 0) 0x7ff640138170 con 0x7ff64014be90 2024-03-25T19:34:25.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.057+0000 7ff63d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:25.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.057+0000 7ff63d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:25.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.057+0000 7ff63d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:25.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.057+0000 7ff63d7fa640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:34:25.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.057+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff640137e40 con 0x7ff64014be90 2024-03-25T19:34:25.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.059+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464187029 0 0) 0x7ff6401324b0 con 0x7ff6400b4d60 2024-03-25T19:34:25.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.059+0000 7ff63d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:25.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.059+0000 7ff63d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:25.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.059+0000 7ff63d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:25.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.059+0000 7ff63d7fa640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:34:25.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.059+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff640138170 con 0x7ff6400b4d60 2024-03-25T19:34:25.062 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.062+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 311550811 0 0) 0x7ff624004eb0 con 0x7ff6400b1730 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.062+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6401324b0 con 0x7ff6400b1730 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.062+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff634001b70 con 0x7ff6400b1730 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.062+0000 7ff63d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.062+0000 7ff63d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.062+0000 7ff63d7fa640 10 monclient(hunting): dump: 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:25.063 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:25.064 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:25.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.062+0000 7ff63d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.070+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2944245974 0 0) 0x7ff640137e40 con 0x7ff64014be90 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.070+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff624004eb0 con 0x7ff64014be90 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.070+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff638001b10 con 0x7ff64014be90 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.070+0000 7ff63d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.070+0000 7ff63d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.070+0000 7ff63d7fa640 10 monclient(hunting): dump: 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:25.070 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:25.071 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:25.071 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:25.071 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:25.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.079+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4227942158 0 0) 0x7ff640138170 con 0x7ff6400b4d60 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.079+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff640137e40 con 0x7ff6400b4d60 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.079+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff62c0018a0 con 0x7ff6400b4d60 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.079+0000 7ff63d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.079+0000 7ff63d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.079+0000 7ff63d7fa640 10 monclient(hunting): dump: 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:25.080 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:25.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2311689914 0 0) 0x7ff624004eb0 con 0x7ff64014be90 2024-03-25T19:34:25.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 >> v1:172.21.15.19:6790/0 conn(0x7ff6400b4d60 legacy=0x7ff640131da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 >> v1:172.21.15.188:6789/0 conn(0x7ff6400b1730 legacy=0x7ff640138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff63d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:34:25.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff63d7fa640 1 monclient: found mon.a 2024-03-25T19:34:25.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff63d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:25.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff63d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff640132640 con 0x7ff64014be90 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff63d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff63d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff63d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff646e55640 5 monclient: authenticate success, global_id 4473 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff646e55640 10 monclient: _renew_subs 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff646e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.087+0000 7ff646e55640 1 -- 172.21.15.19:0/2351923287 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff640145260 con 0x7ff64014be90 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.088+0000 7ff646e55640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.088+0000 7ff646e55640 10 monclient: _renew_subs 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.088+0000 7ff646e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.088+0000 7ff646e55640 1 -- 172.21.15.19:0/2351923287 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64013b670 con 0x7ff64014be90 2024-03-25T19:34:25.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.088+0000 7ff646e55640 1 librados: init done 2024-03-25T19:34:25.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.088+0000 7ff646e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:25.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.088+0000 7ff646e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:25.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.088+0000 7ff646e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:34:25.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.088+0000 7ff646e55640 1 -- 172.21.15.19:0/2351923287 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff640137fd0 con 0x7ff64014be90 2024-03-25T19:34:25.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.313+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff638002000 con 0x7ff64014be90 2024-03-25T19:34:25.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.313+0000 7ff63d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:25.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.313+0000 7ff63effd640 4 set_mon_vals no callback set 2024-03-25T19:34:25.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.416+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff638003970 con 0x7ff64014be90 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.416+0000 7ff63d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.417+0000 7ff63d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.417+0000 7ff63d7fa640 10 monclient: dump: 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:25.417 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:25.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.417+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7ff638003b50 con 0x7ff64014be90 2024-03-25T19:34:25.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.418+0000 7ff644bf1640 10 monclient: get_auth_request con 0x7ff62406e0c0 auth_method 0 2024-03-25T19:34:25.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.418+0000 7ff63d7fa640 10 client.4473.objecter ms_handle_connect 0x7ff62406e0c0 2024-03-25T19:34:25.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.426+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7ff638053610 con 0x7ff64014be90 2024-03-25T19:34:25.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.426+0000 7ff63d7fa640 10 client.4473.objecter ms_dispatch 0x7ff640000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:25.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.426+0000 7ff63d7fa640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:25.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.426+0000 7ff63d7fa640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:25.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.426+0000 7ff63d7fa640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-25T19:34:25.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.426+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff63804ef50 con 0x7ff64014be90 2024-03-25T19:34:25.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.426+0000 7ff63d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:25.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.426+0000 7ff63d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:25.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.678+0000 7ff646e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:25.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.678+0000 7ff646e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:25.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.678+0000 7ff646e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:34:25.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.678+0000 7ff646e55640 1 -- 172.21.15.19:0/2351923287 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff640138300 con 0x7ff64014be90 2024-03-25T19:34:25.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.706+0000 7ff63d7fa640 1 -- 172.21.15.19:0/2351923287 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7ff638051050 con 0x7ff64014be90 2024-03-25T19:34:25.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.706+0000 7ff63d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:25.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.706+0000 7ff63d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:25.706 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:34:25.707 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:25.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 10 librados: watch_flush enter 2024-03-25T19:34:25.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 10 librados: watch_flush exit 2024-03-25T19:34:25.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-25T19:34:25.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 10 client.4473.objecter successfully canceled tick 2024-03-25T19:34:25.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 1 -- 172.21.15.19:0/2351923287 >> v1:172.21.15.188:6800/2207506229 conn(0x7ff62406e0c0 legacy=0x7ff624070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 10 monclient: shutdown 2024-03-25T19:34:25.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:25.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 1 -- 172.21.15.19:0/2351923287 >> v1:172.21.15.19:6789/0 conn(0x7ff64014be90 legacy=0x7ff64013af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 1 -- 172.21.15.19:0/2351923287 shutdown_connections 2024-03-25T19:34:25.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 1 -- 172.21.15.19:0/2351923287 >> 172.21.15.19:0/2351923287 conn(0x7ff6400617a0 msgr2=0x7ff64013c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:25.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 1 -- 172.21.15.19:0/2351923287 shutdown_connections 2024-03-25T19:34:25.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 1 -- 172.21.15.19:0/2351923287 wait complete. 2024-03-25T19:34:25.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:25.709+0000 7ff646e55640 1 librados: shutdown 2024-03-25T19:34:25.722 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:32.528 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:32.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.635+0000 7f0b31206640 10 monclient: get_monmap_and_config 2024-03-25T19:34:32.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.635+0000 7f0b31206640 10 monclient: build_initial_monmap 2024-03-25T19:34:32.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.635+0000 7f0b31206640 10 monclient: monmap: 2024-03-25T19:34:32.635 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:32.636 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:32.636 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:34:32.636793+0000 2024-03-25T19:34:32.636 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:34:32.636793+0000 2024-03-25T19:34:32.636 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:32.636 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:32.636 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:32.636 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:34:32.636 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:34:32.636 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:32.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 1 Processor -- start 2024-03-25T19:34:32.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 1 -- start start 2024-03-25T19:34:32.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 10 monclient: init 2024-03-25T19:34:32.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:32.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:32.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 10 monclient(hunting): picked mon.noname-b con 0x7f0b2c0a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:32.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 10 monclient(hunting): picked mon.noname-c con 0x7f0b2c13f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:32.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 10 monclient(hunting): picked mon.noname-a con 0x7f0b2c1430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:32.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c142520 con 0x7f0b2c1430c0 2024-03-25T19:34:32.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c0b28a0 con 0x7f0b2c13f710 2024-03-25T19:34:32.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c0b2a30 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:32.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.636+0000 7f0b31206640 10 monclient(hunting): authenticate will time out at 6325.906700s 2024-03-25T19:34:32.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.637+0000 7f0b2a59c640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f0b2c13f710 0x7f0b2c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:46668/0 (socket says 172.21.15.19:46668) 2024-03-25T19:34:32.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.637+0000 7f0b2ad9d640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f0b2c0a98f0 0x7f0b2c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:60978/0 (socket says 172.21.15.19:60978) 2024-03-25T19:34:32.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.637+0000 7f0b2a59c640 1 -- 172.21.15.19:0/909557150 learned_addr learned my addr 172.21.15.19:0/909557150 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:32.663 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:55.392 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.670 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:55.400 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.702 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:55.431 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.730 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:05:55.433 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.735 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:05:55.438 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.735 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:55.464 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.740+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500219018 0 0) 0x7f0b2c0b28a0 con 0x7f0b2c13f710 2024-03-25T19:34:32.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.740+0000 7f0b29d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.740+0000 7f0b29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.740+0000 7f0b29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.740+0000 7f0b29d9b640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:34:32.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.740+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b14003520 con 0x7f0b2c13f710 2024-03-25T19:34:32.742 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:55.472 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.747+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089281136 0 0) 0x7f0b2c0b2a30 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.747+0000 7f0b29d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.747+0000 7f0b29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.747+0000 7f0b29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.747+0000 7f0b29d9b640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:34:32.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.747+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b2c0b28a0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.750+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920756622 0 0) 0x7f0b2c142520 con 0x7f0b2c1430c0 2024-03-25T19:34:32.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.750+0000 7f0b29d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.750+0000 7f0b29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.750+0000 7f0b29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.750+0000 7f0b29d9b640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:34:32.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.750+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b2c0b2a30 con 0x7f0b2c1430c0 2024-03-25T19:34:32.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2016038375 0 0) 0x7f0b2c0b28a0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 >> v1:172.21.15.188:6789/0 conn(0x7f0b2c13f710 legacy=0x7f0b2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 >> v1:172.21.15.19:6789/0 conn(0x7f0b2c1430c0 legacy=0x7f0b2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:34:32.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 1 monclient: found mon.noname-b 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c0b44c0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 10 monclient: _finish_auth 0 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0b1c001670 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b31206640 5 monclient: authenticate success, global_id 4448 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b31206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b29d9b640 10 monclient: dump: 2024-03-25T19:34:32.765 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:32.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.764+0000 7f0b31206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:32.766 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:05:55.467 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.774+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b1c0017d0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.774+0000 7f0b29d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:32.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.774+0000 7f0b29d9b640 1 -- 172.21.15.19:0/909557150 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0b1c001be0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.774+0000 7f0b29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.774+0000 7f0b29d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:32.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.774+0000 7f0b29d9b640 10 monclient: dump: 2024-03-25T19:34:32.775 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:32.775 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:32.775 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:32.775 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:32.775 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.775 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:32.775 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:32.775 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:32.775 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:32.775 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:32.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.774+0000 7f0b31206640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:32.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.774+0000 7f0b31206640 4 set_mon_vals no callback set 2024-03-25T19:34:32.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.775+0000 7f0b31206640 10 monclient: shutdown 2024-03-25T19:34:32.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.775+0000 7f0b31206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:32.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.775+0000 7f0b31206640 1 -- 172.21.15.19:0/909557150 >> v1:172.21.15.19:6790/0 conn(0x7f0b2c0a98f0 legacy=0x7f0b2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.775+0000 7f0b31206640 1 -- 172.21.15.19:0/909557150 shutdown_connections 2024-03-25T19:34:32.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.775+0000 7f0b31206640 1 -- 172.21.15.19:0/909557150 >> 172.21.15.19:0/909557150 conn(0x7f0b2c0b7ea0 msgr2=0x7f0b2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:32.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.775+0000 7f0b31206640 1 -- 172.21.15.19:0/909557150 shutdown_connections 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.775+0000 7f0b31206640 1 -- 172.21.15.19:0/909557150 wait complete. 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.776+0000 7f0b31206640 10 monclient: build_initial_monmap 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.776+0000 7f0b31206640 10 monclient: monmap: 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.777+0000 7f0b31206640 1 librados: starting msgr at 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.777+0000 7f0b31206640 1 librados: starting objecter 2024-03-25T19:34:32.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.777+0000 7f0b31206640 1 Processor -- start 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.777+0000 7f0b31206640 1 -- start start 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.777+0000 7f0b31206640 1 librados: setting wanted keys 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.777+0000 7f0b31206640 1 librados: calling monclient init 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.777+0000 7f0b31206640 10 monclient: init 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.778+0000 7f0b31206640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.778+0000 7f0b31206640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.778+0000 7f0b31206640 10 monclient(hunting): picked mon.noname-c con 0x7f0b2c0a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.778+0000 7f0b31206640 10 monclient(hunting): picked mon.noname-b con 0x7f0b2c13f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.778+0000 7f0b31206640 10 monclient(hunting): picked mon.noname-a con 0x7f0b2c1430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.778+0000 7f0b31206640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c136e60 con 0x7f0b2c1430c0 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.778+0000 7f0b31206640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c1381a0 con 0x7f0b2c13f710 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.778+0000 7f0b31206640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c1370b0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.778+0000 7f0b31206640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:32.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.778+0000 7f0b31206640 10 monclient(hunting): authenticate will time out at 6326.048643s 2024-03-25T19:34:32.781 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:55.508 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.792+0000 7f0b2a59c640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f0b2c13f710 0x7f0b2c13a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:46702/0 (socket says 172.21.15.19:46702) 2024-03-25T19:34:32.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.793+0000 7f0b2a59c640 1 -- 172.21.15.19:0/147935056 learned_addr learned my addr 172.21.15.19:0/147935056 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:32.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.797+0000 7f0b137fe640 10 client.?.objecter ms_handle_connect 0x7f0b2c13f710 2024-03-25T19:34:32.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.797+0000 7f0b137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:32.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.805+0000 7f0b137fe640 10 client.?.objecter ms_handle_connect 0x7f0b2c0a98f0 2024-03-25T19:34:32.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.805+0000 7f0b137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:32.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.822+0000 7f0b137fe640 10 client.?.objecter ms_handle_connect 0x7f0b2c1430c0 2024-03-25T19:34:32.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.822+0000 7f0b137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:32.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.834+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206777849 0 0) 0x7f0b2c1381a0 con 0x7f0b2c13f710 2024-03-25T19:34:32.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.834+0000 7f0b137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.834+0000 7f0b137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.834+0000 7f0b137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.834+0000 7f0b137fe640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:34:32.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.834+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b18004eb0 con 0x7f0b2c13f710 2024-03-25T19:34:32.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.835+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856502610 0 0) 0x7f0b2c1370b0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.835+0000 7f0b137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.835+0000 7f0b137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.835+0000 7f0b137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.835+0000 7f0b137fe640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:34:32.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.835+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b2c1381a0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.842+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3267873023 0 0) 0x7f0b18004eb0 con 0x7f0b2c13f710 2024-03-25T19:34:32.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.842+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b2c1370b0 con 0x7f0b2c13f710 2024-03-25T19:34:32.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.846+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0b20001930 con 0x7f0b2c13f710 2024-03-25T19:34:32.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.846+0000 7f0b137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.846+0000 7f0b137fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:32.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.846+0000 7f0b137fe640 10 monclient(hunting): dump: 2024-03-25T19:34:32.847 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:32.847 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:32.847 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:32.847 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:32.847 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.847 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:32.847 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:32.847 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:32.847 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:32.847 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:32.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.846+0000 7f0b137fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:32.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.855+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3577596598 0 0) 0x7f0b2c1381a0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.855+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b18004eb0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.855+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0b1c001ae0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.855+0000 7f0b137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.855+0000 7f0b137fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:32.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.855+0000 7f0b137fe640 10 monclient(hunting): dump: 2024-03-25T19:34:32.855 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:32.855 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:32.855 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:32.855 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:32.855 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.856 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:32.856 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:32.856 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:32.856 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:32.856 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:32.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.861+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588618234 0 0) 0x7f0b2c136e60 con 0x7f0b2c1430c0 2024-03-25T19:34:32.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.861+0000 7f0b137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.861+0000 7f0b137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.861+0000 7f0b137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.861+0000 7f0b137fe640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:34:32.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.861+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b2c1381a0 con 0x7f0b2c1430c0 2024-03-25T19:34:32.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.876+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1607297528 0 0) 0x7f0b18004eb0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 >> v1:172.21.15.188:6789/0 conn(0x7f0b2c13f710 legacy=0x7f0b2c13a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 >> v1:172.21.15.19:6789/0 conn(0x7f0b2c1430c0 legacy=0x7f0b2c13ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b137fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:32.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b137fe640 1 monclient: found mon.c 2024-03-25T19:34:32.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:32.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b137fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:32.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c137390 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b137fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:32.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b137fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:32.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:32.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b31206640 5 monclient: authenticate success, global_id 4451 2024-03-25T19:34:32.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b31206640 10 monclient: _renew_subs 2024-03-25T19:34:32.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b31206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:32.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b31206640 1 -- 172.21.15.19:0/147935056 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c1376d0 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b31206640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:32.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b31206640 10 monclient: _renew_subs 2024-03-25T19:34:32.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b31206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:32.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b31206640 1 -- 172.21.15.19:0/147935056 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c135530 con 0x7f0b2c0a98f0 2024-03-25T19:34:32.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.877+0000 7f0b31206640 1 librados: init done 2024-03-25T19:34:32.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.878+0000 7f0b31206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:32.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.878+0000 7f0b31206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:32.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.878+0000 7f0b31206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:32.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:32.878+0000 7f0b31206640 1 -- 172.21.15.19:0/147935056 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2c0b3ef0 con 0x7f0b2c0a98f0 2024-03-25T19:34:33.192 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.192+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b1c001210 con 0x7f0b2c0a98f0 2024-03-25T19:34:33.192 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.192+0000 7f0b137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:33.192 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.192+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0b1c004c30 con 0x7f0b2c0a98f0 2024-03-25T19:34:33.192 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.192+0000 7f0b137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:33.192 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.192+0000 7f0b137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:33.192 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.192+0000 7f0b137fe640 10 monclient: dump: 2024-03-25T19:34:33.192 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:33.193 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:33.193 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:33.193 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:33.193 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:33.193 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:33.193 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:33.193 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:33.193 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:33.193 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:33.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.192+0000 7f0b2959a640 4 set_mon_vals no callback set 2024-03-25T19:34:33.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.193+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f0b1c01ac00 con 0x7f0b2c0a98f0 2024-03-25T19:34:33.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.193+0000 7f0b2a59c640 10 monclient: get_auth_request con 0x7f0b1806dee0 auth_method 0 2024-03-25T19:34:33.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.194+0000 7f0b137fe640 10 client.4451.objecter ms_handle_connect 0x7f0b1806dee0 2024-03-25T19:34:33.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.201+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f0b1c0547e0 con 0x7f0b2c0a98f0 2024-03-25T19:34:33.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.201+0000 7f0b137fe640 10 client.4451.objecter ms_dispatch 0x7f0b2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:33.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.201+0000 7f0b137fe640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:33.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.201+0000 7f0b137fe640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:33.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.201+0000 7f0b137fe640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-25T19:34:33.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.201+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b1c050120 con 0x7f0b2c0a98f0 2024-03-25T19:34:33.202 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.201+0000 7f0b137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:33.202 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.201+0000 7f0b137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:33.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.446+0000 7f0b31206640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:33.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.446+0000 7f0b31206640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:33.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.446+0000 7f0b31206640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:33.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.446+0000 7f0b31206640 1 -- 172.21.15.19:0/147935056 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b2c1357e0 con 0x7f0b2c0a98f0 2024-03-25T19:34:33.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.473+0000 7f0b137fe640 1 -- 172.21.15.19:0/147935056 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f0b1c052220 con 0x7f0b2c0a98f0 2024-03-25T19:34:33.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.473+0000 7f0b137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:33.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.473+0000 7f0b137fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:33.473 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:34:33.473 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:33.476 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.475+0000 7f0b31206640 10 librados: watch_flush enter 2024-03-25T19:34:33.476 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.475+0000 7f0b31206640 10 librados: watch_flush exit 2024-03-25T19:34:33.476 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.475+0000 7f0b31206640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-25T19:34:33.476 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.475+0000 7f0b31206640 10 client.4451.objecter successfully canceled tick 2024-03-25T19:34:33.476 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.476+0000 7f0b31206640 1 -- 172.21.15.19:0/147935056 >> v1:172.21.15.188:6800/2207506229 conn(0x7f0b1806dee0 legacy=0x7f0b180703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:33.476 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.476+0000 7f0b31206640 10 monclient: shutdown 2024-03-25T19:34:33.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.476+0000 7f0b31206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:33.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.476+0000 7f0b31206640 1 -- 172.21.15.19:0/147935056 >> v1:172.21.15.19:6790/0 conn(0x7f0b2c0a98f0 legacy=0x7f0b2c137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:33.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.476+0000 7f0b31206640 1 -- 172.21.15.19:0/147935056 shutdown_connections 2024-03-25T19:34:33.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.476+0000 7f0b31206640 1 -- 172.21.15.19:0/147935056 >> 172.21.15.19:0/147935056 conn(0x7f0b2c0b7ea0 msgr2=0x7f0b2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:33.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.476+0000 7f0b31206640 1 -- 172.21.15.19:0/147935056 shutdown_connections 2024-03-25T19:34:33.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.476+0000 7f0b31206640 1 -- 172.21.15.19:0/147935056 wait complete. 2024-03-25T19:34:33.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:33.476+0000 7f0b31206640 1 librados: shutdown 2024-03-25T19:34:33.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:40.296 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:40.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.413+0000 7f5953aa8640 10 monclient: get_monmap_and_config 2024-03-25T19:34:40.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.413+0000 7f5953aa8640 10 monclient: build_initial_monmap 2024-03-25T19:34:40.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.413+0000 7f5953aa8640 10 monclient: monmap: 2024-03-25T19:34:40.414 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:40.414 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:40.414 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:34:40.415013+0000 2024-03-25T19:34:40.414 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:34:40.415013+0000 2024-03-25T19:34:40.414 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:40.414 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:40.414 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:40.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.414+0000 7f5953aa8640 1 Processor -- start 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.414+0000 7f5953aa8640 1 -- start start 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.414+0000 7f5953aa8640 10 monclient: init 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.415+0000 7f5953aa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.415+0000 7f5953aa8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.415+0000 7f5953aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f594c0a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.415+0000 7f5953aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f594c13f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:40.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.415+0000 7f5953aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f594c1430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:40.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.415+0000 7f5953aa8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c142520 con 0x7f594c13f710 2024-03-25T19:34:40.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.415+0000 7f5953aa8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c0b28a0 con 0x7f594c1430c0 2024-03-25T19:34:40.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.415+0000 7f5953aa8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c0b2a30 con 0x7f594c0a98f0 2024-03-25T19:34:40.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.415+0000 7f5953aa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:40.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.415+0000 7f5953aa8640 10 monclient(hunting): authenticate will time out at 6333.685458s 2024-03-25T19:34:40.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.420+0000 7f5952045640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f594c1430c0 0x7f594c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:58822/0 (socket says 172.21.15.19:58822) 2024-03-25T19:34:40.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.420+0000 7f5952045640 1 -- 172.21.15.19:0/1110573440 learned_addr learned my addr 172.21.15.19:0/1110573440 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:40.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.427+0000 7f5950842640 1 -- 172.21.15.19:0/1110573440 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247074556 0 0) 0x7f594c0b28a0 con 0x7f594c1430c0 2024-03-25T19:34:40.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.427+0000 7f5950842640 10 monclient(hunting): _negotiate 2024-03-25T19:34:40.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.427+0000 7f5950842640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:40.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.427+0000 7f5950842640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:40.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.427+0000 7f5950842640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:34:40.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.427+0000 7f5950842640 1 -- 172.21.15.19:0/1110573440 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5938003520 con 0x7f594c1430c0 2024-03-25T19:34:40.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.428+0000 7f5950842640 1 -- 172.21.15.19:0/1110573440 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663641334 0 0) 0x7f594c142520 con 0x7f594c13f710 2024-03-25T19:34:40.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.428+0000 7f5950842640 10 monclient(hunting): _negotiate 2024-03-25T19:34:40.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.428+0000 7f5950842640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:40.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.428+0000 7f5950842640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:40.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.428+0000 7f5950842640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:34:40.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.428+0000 7f5950842640 1 -- 172.21.15.19:0/1110573440 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f594c0b28a0 con 0x7f594c13f710 2024-03-25T19:34:40.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 1 -- 172.21.15.19:0/1110573440 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3026740435 0 0) 0x7f594c0b28a0 con 0x7f594c13f710 2024-03-25T19:34:40.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 1 -- 172.21.15.19:0/1110573440 >> v1:172.21.15.19:6790/0 conn(0x7f594c0a98f0 legacy=0x7f594c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 1 -- 172.21.15.19:0/1110573440 >> v1:172.21.15.188:6789/0 conn(0x7f594c1430c0 legacy=0x7f594c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 10 monclient: _finish_hunting 0 2024-03-25T19:34:40.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 1 monclient: found mon.noname-a 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 1 -- 172.21.15.19:0/1110573440 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c0b44c0 con 0x7f594c13f710 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 10 monclient: _finish_auth 0 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 10 monclient: _check_auth_tickets 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 1 -- 172.21.15.19:0/1110573440 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f593c001670 con 0x7f594c13f710 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5953aa8640 5 monclient: authenticate success, global_id 4482 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5953aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5950842640 10 monclient: dump: 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:40.437 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:40.438 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:40.438 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:40.438 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:40.438 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:40.438 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:40.438 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:40.438 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:40.438 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:40.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.436+0000 7f5953aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:40.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.449+0000 7f5950842640 1 -- 172.21.15.19:0/1110573440 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f593c0017d0 con 0x7f594c13f710 2024-03-25T19:34:40.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.449+0000 7f5950842640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:40.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.449+0000 7f5950842640 1 -- 172.21.15.19:0/1110573440 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f593c001ba0 con 0x7f594c13f710 2024-03-25T19:34:40.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.449+0000 7f5953aa8640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:40.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.449+0000 7f5953aa8640 4 set_mon_vals no callback set 2024-03-25T19:34:40.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.449+0000 7f5950842640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:40.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.449+0000 7f5950842640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:40.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.449+0000 7f5950842640 10 monclient: dump: 2024-03-25T19:34:40.450 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:40.450 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:40.450 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:40.450 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:40.450 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:40.450 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:40.450 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:40.450 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:40.450 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:40.450 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:40.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.450+0000 7f5953aa8640 10 monclient: shutdown 2024-03-25T19:34:40.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.450+0000 7f5953aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:40.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.450+0000 7f5953aa8640 1 -- 172.21.15.19:0/1110573440 >> v1:172.21.15.19:6789/0 conn(0x7f594c13f710 legacy=0x7f594c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.450+0000 7f5953aa8640 1 -- 172.21.15.19:0/1110573440 shutdown_connections 2024-03-25T19:34:40.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.450+0000 7f5953aa8640 1 -- 172.21.15.19:0/1110573440 >> 172.21.15.19:0/1110573440 conn(0x7f594c0b7ea0 msgr2=0x7f594c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:40.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.450+0000 7f5953aa8640 1 -- 172.21.15.19:0/1110573440 shutdown_connections 2024-03-25T19:34:40.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.450+0000 7f5953aa8640 1 -- 172.21.15.19:0/1110573440 wait complete. 2024-03-25T19:34:40.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.451+0000 7f5953aa8640 10 monclient: build_initial_monmap 2024-03-25T19:34:40.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.451+0000 7f5953aa8640 10 monclient: monmap: 2024-03-25T19:34:40.452 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:40.452 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.451+0000 7f5953aa8640 1 librados: starting msgr at 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.451+0000 7f5953aa8640 1 librados: starting objecter 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.452+0000 7f5953aa8640 1 Processor -- start 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.452+0000 7f5953aa8640 1 -- start start 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.452+0000 7f5953aa8640 1 librados: setting wanted keys 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.452+0000 7f5953aa8640 1 librados: calling monclient init 2024-03-25T19:34:40.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.452+0000 7f5953aa8640 10 monclient: init 2024-03-25T19:34:40.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.452+0000 7f5953aa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:40.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.452+0000 7f5953aa8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:40.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.452+0000 7f5953aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f594c0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:40.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.452+0000 7f5953aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f594c13f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:40.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.453+0000 7f5953aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f594c1430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:40.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.453+0000 7f5953aa8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c1386e0 con 0x7f594c0a98f0 2024-03-25T19:34:40.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.453+0000 7f5953aa8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c137e80 con 0x7f594c13f710 2024-03-25T19:34:40.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.453+0000 7f5953aa8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c1398f0 con 0x7f594c1430c0 2024-03-25T19:34:40.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.453+0000 7f5953aa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:40.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.453+0000 7f5953aa8640 10 monclient(hunting): authenticate will time out at 6333.723075s 2024-03-25T19:34:40.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.462+0000 7f5951043640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f594c13f710 0x7f594c1d7490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:58836/0 (socket says 172.21.15.19:58836) 2024-03-25T19:34:40.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.462+0000 7f5951043640 1 -- 172.21.15.19:0/2522047298 learned_addr learned my addr 172.21.15.19:0/2522047298 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:40.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.465+0000 7f59427fc640 10 client.?.objecter ms_handle_connect 0x7f594c13f710 2024-03-25T19:34:40.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.466+0000 7f59427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:40.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.467+0000 7f59427fc640 10 client.?.objecter ms_handle_connect 0x7f594c1430c0 2024-03-25T19:34:40.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.467+0000 7f59427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:40.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.468+0000 7f59427fc640 10 client.?.objecter ms_handle_connect 0x7f594c0a98f0 2024-03-25T19:34:40.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.468+0000 7f59427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:40.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.479+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149123555 0 0) 0x7f594c1398f0 con 0x7f594c1430c0 2024-03-25T19:34:40.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.479+0000 7f59427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:40.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.479+0000 7f59427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:40.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.479+0000 7f59427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:40.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.479+0000 7f59427fc640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:34:40.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.479+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5930004eb0 con 0x7f594c1430c0 2024-03-25T19:34:40.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.483+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556320671 0 0) 0x7f594c1386e0 con 0x7f594c0a98f0 2024-03-25T19:34:40.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.483+0000 7f59427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:40.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.483+0000 7f59427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:40.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.483+0000 7f59427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:40.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.483+0000 7f59427fc640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:34:40.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.483+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f594c1398f0 con 0x7f594c0a98f0 2024-03-25T19:34:40.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.485+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992310063 0 0) 0x7f594c137e80 con 0x7f594c13f710 2024-03-25T19:34:40.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.485+0000 7f59427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:40.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.485+0000 7f59427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:40.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.485+0000 7f59427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:40.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.485+0000 7f59427fc640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:34:40.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.485+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f594c1386e0 con 0x7f594c13f710 2024-03-25T19:34:40.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.488+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1417287199 0 0) 0x7f5930004eb0 con 0x7f594c1430c0 2024-03-25T19:34:40.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.488+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f594c137e80 con 0x7f594c1430c0 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.488+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5944001b90 con 0x7f594c1430c0 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.488+0000 7f59427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.488+0000 7f59427fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.488+0000 7f59427fc640 10 monclient(hunting): dump: 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:40.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.488+0000 7f59427fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:40.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.495+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1438096995 0 0) 0x7f594c1386e0 con 0x7f594c13f710 2024-03-25T19:34:40.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.495+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5930004eb0 con 0x7f594c13f710 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.495+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f593c001ba0 con 0x7f594c13f710 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.495+0000 7f59427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.495+0000 7f59427fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.495+0000 7f59427fc640 10 monclient(hunting): dump: 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:40.496 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:40.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 542264116 0 0) 0x7f594c137e80 con 0x7f594c1430c0 2024-03-25T19:34:40.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 >> v1:172.21.15.188:6789/0 conn(0x7f594c13f710 legacy=0x7f594c1d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 >> v1:172.21.15.19:6789/0 conn(0x7f594c0a98f0 legacy=0x7f594c137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f59427fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:40.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f59427fc640 1 monclient: found mon.c 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f59427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f59427fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c1380a0 con 0x7f594c1430c0 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f59427fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f59427fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f59427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f5953aa8640 5 monclient: authenticate success, global_id 4457 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f5953aa8640 10 monclient: _renew_subs 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f5953aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f5953aa8640 1 -- 172.21.15.19:0/2522047298 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594c138350 con 0x7f594c1430c0 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f5953aa8640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f5953aa8640 10 monclient: _renew_subs 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f5953aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f5953aa8640 1 -- 172.21.15.19:0/2522047298 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f594c1d7d80 con 0x7f594c1430c0 2024-03-25T19:34:40.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.504+0000 7f5953aa8640 1 librados: init done 2024-03-25T19:34:40.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.505+0000 7f5953aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:40.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.505+0000 7f5953aa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:40.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.505+0000 7f5953aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:40.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.505+0000 7f5953aa8640 1 -- 172.21.15.19:0/2522047298 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594c1d8030 con 0x7f594c1430c0 2024-03-25T19:34:40.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.822+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5944001fe0 con 0x7f594c1430c0 2024-03-25T19:34:40.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.822+0000 7f59427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:40.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.822+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5944004d00 con 0x7f594c1430c0 2024-03-25T19:34:40.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.822+0000 7f59427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:40.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.822+0000 7f59427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:40.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.822+0000 7f59427fc640 10 monclient: dump: 2024-03-25T19:34:40.823 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:40.823 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:40.823 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:40.823 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:40.823 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:40.823 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:40.823 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:40.823 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:40.823 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:40.823 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:40.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.822+0000 7f5943fff640 4 set_mon_vals no callback set 2024-03-25T19:34:40.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.823+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f5944004f20 con 0x7f594c1430c0 2024-03-25T19:34:40.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.823+0000 7f5951844640 10 monclient: get_auth_request con 0x7f593006df20 auth_method 0 2024-03-25T19:34:40.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.824+0000 7f59427fc640 10 client.4457.objecter ms_handle_connect 0x7f593006df20 2024-03-25T19:34:40.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.832+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f5944054ae0 con 0x7f594c1430c0 2024-03-25T19:34:40.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.832+0000 7f59427fc640 10 client.4457.objecter ms_dispatch 0x7f594c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:40.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.832+0000 7f59427fc640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:40.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.832+0000 7f59427fc640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:40.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.832+0000 7f59427fc640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-25T19:34:40.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.832+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5944050420 con 0x7f594c1430c0 2024-03-25T19:34:40.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.832+0000 7f59427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:40.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:40.833+0000 7f59427fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:41.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.065+0000 7f5953aa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:41.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.065+0000 7f5953aa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:41.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.065+0000 7f5953aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:34:41.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.065+0000 7f5953aa8640 1 -- 172.21.15.19:0/2522047298 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f594c0627b0 con 0x7f594c1430c0 2024-03-25T19:34:41.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.102+0000 7f59427fc640 1 -- 172.21.15.19:0/2522047298 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f5944052520 con 0x7f594c1430c0 2024-03-25T19:34:41.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.102+0000 7f59427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:41.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.102+0000 7f59427fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:41.102 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:34:41.102 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:41.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.104+0000 7f5953aa8640 10 librados: watch_flush enter 2024-03-25T19:34:41.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.104+0000 7f5953aa8640 10 librados: watch_flush exit 2024-03-25T19:34:41.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.104+0000 7f5953aa8640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-25T19:34:41.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.104+0000 7f5953aa8640 10 client.4457.objecter successfully canceled tick 2024-03-25T19:34:41.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.104+0000 7f5953aa8640 1 -- 172.21.15.19:0/2522047298 >> v1:172.21.15.188:6800/2207506229 conn(0x7f593006df20 legacy=0x7f59300703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:41.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.104+0000 7f5953aa8640 10 monclient: shutdown 2024-03-25T19:34:41.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.104+0000 7f5953aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:41.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.104+0000 7f5953aa8640 1 -- 172.21.15.19:0/2522047298 >> v1:172.21.15.19:6790/0 conn(0x7f594c1430c0 legacy=0x7f594c1e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:41.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.105+0000 7f5953aa8640 1 -- 172.21.15.19:0/2522047298 shutdown_connections 2024-03-25T19:34:41.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.105+0000 7f5953aa8640 1 -- 172.21.15.19:0/2522047298 >> 172.21.15.19:0/2522047298 conn(0x7f594c0b7ea0 msgr2=0x7f594c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:41.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.105+0000 7f5953aa8640 1 -- 172.21.15.19:0/2522047298 shutdown_connections 2024-03-25T19:34:41.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.105+0000 7f5953aa8640 1 -- 172.21.15.19:0/2522047298 wait complete. 2024-03-25T19:34:41.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:41.105+0000 7f5953aa8640 1 librados: shutdown 2024-03-25T19:34:41.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:47.921 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:48.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.042+0000 7fdd3ab0e640 10 monclient: get_monmap_and_config 2024-03-25T19:34:48.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.042+0000 7fdd3ab0e640 10 monclient: build_initial_monmap 2024-03-25T19:34:48.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.042+0000 7fdd3ab0e640 10 monclient: monmap: 2024-03-25T19:34:48.042 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:48.042 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:48.042 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:34:48.043351+0000 2024-03-25T19:34:48.042 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:34:48.043351+0000 2024-03-25T19:34:48.042 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:48.042 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:48.042 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:48.042 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:34:48.042 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:34:48.043 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:48.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.042+0000 7fdd3ab0e640 1 Processor -- start 2024-03-25T19:34:48.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.043+0000 7fdd3ab0e640 1 -- start start 2024-03-25T19:34:48.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.043+0000 7fdd3ab0e640 10 monclient: init 2024-03-25T19:34:48.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.043+0000 7fdd3ab0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:48.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.043+0000 7fdd3ab0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:48.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.043+0000 7fdd3ab0e640 10 monclient(hunting): picked mon.noname-b con 0x7fdd340b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:48.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.043+0000 7fdd3ab0e640 10 monclient(hunting): picked mon.noname-c con 0x7fdd340b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:48.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.043+0000 7fdd3ab0e640 10 monclient(hunting): picked mon.noname-a con 0x7fdd3414be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:48.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.043+0000 7fdd3ab0e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd340af940 con 0x7fdd3414be90 2024-03-25T19:34:48.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.044+0000 7fdd3ab0e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd340b0a70 con 0x7fdd340b1730 2024-03-25T19:34:48.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.044+0000 7fdd3ab0e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd340b0d50 con 0x7fdd340b4d60 2024-03-25T19:34:48.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.044+0000 7fdd3ab0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:48.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.044+0000 7fdd3ab0e640 10 monclient(hunting): authenticate will time out at 6341.313964s 2024-03-25T19:34:48.053 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.053+0000 7fdd33fff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fdd340b1730 0x7fdd340b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:35268/0 (socket says 172.21.15.19:35268) 2024-03-25T19:34:48.053 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.053+0000 7fdd33fff640 1 -- 172.21.15.19:0/3956803463 learned_addr learned my addr 172.21.15.19:0/3956803463 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:48.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.058+0000 7fdd33fff640 1 -- 172.21.15.19:0/3956803463 >> v1:172.21.15.188:6789/0 conn(0x7fdd340b1730 legacy=0x7fdd340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T19:34:48.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.058+0000 7fdd33fff640 1 -- 172.21.15.19:0/3956803463 >> v1:172.21.15.188:6789/0 conn(0x7fdd340b1730 legacy=0x7fdd340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:34:48.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.058+0000 7fdd33fff640 1 --1- 172.21.15.19:0/3956803463 >> v1:172.21.15.188:6789/0 conn(0x7fdd340b1730 0x7fdd340b1be0 :-1 s=OPENED pgs=41642 cs=1 l=1).handle_message read tag failed 2024-03-25T19:34:48.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.058+0000 7fdd33fff640 1 --1- 172.21.15.19:0/3956803463 >> v1:172.21.15.188:6789/0 conn(0x7fdd340b1730 0x7fdd340b1be0 :-1 s=OPENED pgs=41642 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:34:48.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.059+0000 7fdd337fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.188:6789/0 2024-03-25T19:34:48.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.067+0000 7fdd337fe640 1 -- 172.21.15.19:0/3956803463 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720796172 0 0) 0x7fdd340b0d50 con 0x7fdd340b4d60 2024-03-25T19:34:48.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.067+0000 7fdd337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:48.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.067+0000 7fdd337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:48.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.067+0000 7fdd337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:48.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.067+0000 7fdd337fe640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:34:48.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.067+0000 7fdd337fe640 1 -- 172.21.15.19:0/3956803463 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd24003520 con 0x7fdd340b4d60 2024-03-25T19:34:48.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 1 -- 172.21.15.19:0/3956803463 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869727769 0 0) 0x7fdd24003520 con 0x7fdd340b4d60 2024-03-25T19:34:48.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 1 -- 172.21.15.19:0/3956803463 >> v1:172.21.15.188:6789/0 conn(0x7fdd340b1730 legacy=0x7fdd340b1be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 1 -- 172.21.15.19:0/3956803463 >> v1:172.21.15.19:6789/0 conn(0x7fdd3414be90 legacy=0x7fdd3414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 1 monclient: found mon.noname-b 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 1 -- 172.21.15.19:0/3956803463 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd340b1030 con 0x7fdd340b4d60 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 1 -- 172.21.15.19:0/3956803463 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fdd28001690 con 0x7fdd340b4d60 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:48.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd337fe640 10 monclient: dump: 2024-03-25T19:34:48.081 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:48.081 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:48.081 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:48.081 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:48.081 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.081 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:48.081 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:48.081 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:48.081 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:48.081 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:48.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd3ab0e640 5 monclient: authenticate success, global_id 4460 2024-03-25T19:34:48.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.079+0000 7fdd3ab0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.093+0000 7fdd337fe640 1 -- 172.21.15.19:0/3956803463 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd280017f0 con 0x7fdd340b4d60 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.093+0000 7fdd337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.093+0000 7fdd337fe640 1 -- 172.21.15.19:0/3956803463 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fdd28001c40 con 0x7fdd340b4d60 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.093+0000 7fdd337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.094+0000 7fdd337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.094+0000 7fdd337fe640 10 monclient: dump: 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:48.094 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:48.095 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:48.095 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:48.095 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:48.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.094+0000 7fdd3ab0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:48.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.094+0000 7fdd3ab0e640 4 set_mon_vals no callback set 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.094+0000 7fdd3ab0e640 10 monclient: shutdown 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.094+0000 7fdd3ab0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.094+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/3956803463 >> v1:172.21.15.19:6790/0 conn(0x7fdd340b4d60 legacy=0x7fdd3413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.094+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/3956803463 shutdown_connections 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.094+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/3956803463 >> 172.21.15.19:0/3956803463 conn(0x7fdd340617a0 msgr2=0x7fdd34061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.094+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/3956803463 shutdown_connections 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.094+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/3956803463 wait complete. 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.095+0000 7fdd3ab0e640 10 monclient: build_initial_monmap 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.095+0000 7fdd3ab0e640 10 monclient: monmap: 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:34:48.097 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.095+0000 7fdd3ab0e640 1 librados: starting msgr at 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.095+0000 7fdd3ab0e640 1 librados: starting objecter 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.095+0000 7fdd3ab0e640 1 Processor -- start 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.095+0000 7fdd3ab0e640 1 -- start start 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.095+0000 7fdd3ab0e640 1 librados: setting wanted keys 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.095+0000 7fdd3ab0e640 1 librados: calling monclient init 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.095+0000 7fdd3ab0e640 10 monclient: init 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.096+0000 7fdd3ab0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.096+0000 7fdd3ab0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.096+0000 7fdd3ab0e640 10 monclient(hunting): picked mon.noname-a con 0x7fdd340b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.096+0000 7fdd3ab0e640 10 monclient(hunting): picked mon.noname-b con 0x7fdd340b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:48.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.096+0000 7fdd3ab0e640 10 monclient(hunting): picked mon.noname-c con 0x7fdd3414be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:48.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.096+0000 7fdd3ab0e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd34138010 con 0x7fdd340b1730 2024-03-25T19:34:48.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.096+0000 7fdd3ab0e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd341382f0 con 0x7fdd340b4d60 2024-03-25T19:34:48.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.096+0000 7fdd3ab0e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd34136d90 con 0x7fdd3414be90 2024-03-25T19:34:48.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.096+0000 7fdd3ab0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:48.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.096+0000 7fdd3ab0e640 10 monclient(hunting): authenticate will time out at 6341.366635s 2024-03-25T19:34:48.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.109+0000 7fdd33fff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fdd340b4d60 0x7fdd34143900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:35298/0 (socket says 172.21.15.19:35298) 2024-03-25T19:34:48.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.109+0000 7fdd33fff640 1 -- 172.21.15.19:0/1025843622 learned_addr learned my addr 172.21.15.19:0/1025843622 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:48.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.111+0000 7fdd317fa640 10 client.?.objecter ms_handle_connect 0x7fdd340b4d60 2024-03-25T19:34:48.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.111+0000 7fdd317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:48.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.115+0000 7fdd317fa640 10 client.?.objecter ms_handle_connect 0x7fdd340b1730 2024-03-25T19:34:48.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.115+0000 7fdd317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:48.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.119+0000 7fdd317fa640 10 client.?.objecter ms_handle_connect 0x7fdd3414be90 2024-03-25T19:34:48.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.119+0000 7fdd317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:48.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.120+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031978342 0 0) 0x7fdd341382f0 con 0x7fdd340b4d60 2024-03-25T19:34:48.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.120+0000 7fdd317fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:48.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.120+0000 7fdd317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:48.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.120+0000 7fdd317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:48.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.120+0000 7fdd317fa640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:34:48.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.120+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd18004eb0 con 0x7fdd340b4d60 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.135+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343880300 0 0) 0x7fdd34138010 con 0x7fdd340b1730 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.135+0000 7fdd317fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.135+0000 7fdd317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.135+0000 7fdd317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.135+0000 7fdd317fa640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:34:48.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.135+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd341382f0 con 0x7fdd340b1730 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.136+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1052965378 0 0) 0x7fdd18004eb0 con 0x7fdd340b4d60 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.137+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd34138010 con 0x7fdd340b4d60 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.137+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fdd200019f0 con 0x7fdd340b4d60 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.137+0000 7fdd317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.137+0000 7fdd317fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.137+0000 7fdd317fa640 10 monclient(hunting): dump: 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:48.137 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:48.138 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:48.138 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:48.138 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:48.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.137+0000 7fdd317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:48.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.143+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818099532 0 0) 0x7fdd34136d90 con 0x7fdd3414be90 2024-03-25T19:34:48.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.143+0000 7fdd317fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:48.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.143+0000 7fdd317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:48.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.143+0000 7fdd317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:48.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.143+0000 7fdd317fa640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:34:48.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.143+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd18004eb0 con 0x7fdd3414be90 2024-03-25T19:34:48.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.156+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3023072524 0 0) 0x7fdd341382f0 con 0x7fdd340b1730 2024-03-25T19:34:48.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.156+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd34136d90 con 0x7fdd340b1730 2024-03-25T19:34:48.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.156+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fdd28001b00 con 0x7fdd340b1730 2024-03-25T19:34:48.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.156+0000 7fdd317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.156+0000 7fdd317fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:48.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.156+0000 7fdd317fa640 10 monclient(hunting): dump: 2024-03-25T19:34:48.156 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:48.156 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:48.156 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 622283057 0 0) 0x7fdd34138010 con 0x7fdd340b4d60 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 >> v1:172.21.15.19:6790/0 conn(0x7fdd3414be90 legacy=0x7fdd341427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 >> v1:172.21.15.19:6789/0 conn(0x7fdd340b1730 legacy=0x7fdd34135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd317fa640 10 monclient: _finish_hunting 0 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd317fa640 1 monclient: found mon.b 2024-03-25T19:34:48.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:48.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd317fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:48.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd34144010 con 0x7fdd340b4d60 2024-03-25T19:34:48.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd317fa640 10 monclient: _finish_auth 0 2024-03-25T19:34:48.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd317fa640 10 monclient: _check_auth_tickets 2024-03-25T19:34:48.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:48.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd3ab0e640 5 monclient: authenticate success, global_id 4483 2024-03-25T19:34:48.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd3ab0e640 10 monclient: _renew_subs 2024-03-25T19:34:48.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd3ab0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:48.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/1025843622 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd341372b0 con 0x7fdd340b4d60 2024-03-25T19:34:48.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd3ab0e640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:48.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd3ab0e640 10 monclient: _renew_subs 2024-03-25T19:34:48.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd3ab0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:48.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/1025843622 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd341369c0 con 0x7fdd340b4d60 2024-03-25T19:34:48.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.162+0000 7fdd3ab0e640 1 librados: init done 2024-03-25T19:34:48.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.163+0000 7fdd3ab0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:48.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.163+0000 7fdd3ab0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:48.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.163+0000 7fdd3ab0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:48.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.163+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/1025843622 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd34136f20 con 0x7fdd340b4d60 2024-03-25T19:34:48.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.452+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd200047e0 con 0x7fdd340b4d60 2024-03-25T19:34:48.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.452+0000 7fdd317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:48.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.452+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fdd20004bf0 con 0x7fdd340b4d60 2024-03-25T19:34:48.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.452+0000 7fdd317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:48.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.452+0000 7fdd317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:48.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.452+0000 7fdd317fa640 10 monclient: dump: 2024-03-25T19:34:48.453 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:48.453 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:48.453 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:48.453 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:48.453 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:48.454 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:48.454 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:48.454 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:48.454 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:48.454 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:48.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.452+0000 7fdd32ffd640 4 set_mon_vals no callback set 2024-03-25T19:34:48.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.453+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fdd20004e10 con 0x7fdd340b4d60 2024-03-25T19:34:48.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.454+0000 7fdd388aa640 10 monclient: get_auth_request con 0x7fdd1806dfb0 auth_method 0 2024-03-25T19:34:48.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.454+0000 7fdd317fa640 10 client.4483.objecter ms_handle_connect 0x7fdd1806dfb0 2024-03-25T19:34:48.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.457+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fdd200548a0 con 0x7fdd340b4d60 2024-03-25T19:34:48.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.457+0000 7fdd317fa640 10 client.4483.objecter ms_dispatch 0x7fdd34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:48.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.457+0000 7fdd317fa640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:48.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.457+0000 7fdd317fa640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:48.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.457+0000 7fdd317fa640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T19:34:48.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.458+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd200503b0 con 0x7fdd340b4d60 2024-03-25T19:34:48.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.458+0000 7fdd317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:48.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.458+0000 7fdd317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:48.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.716+0000 7fdd3ab0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:48.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.716+0000 7fdd3ab0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:48.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.716+0000 7fdd3ab0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:34:48.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.716+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/1025843622 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd340627b0 con 0x7fdd340b4d60 2024-03-25T19:34:48.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.754+0000 7fdd317fa640 1 -- 172.21.15.19:0/1025843622 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fdd200522e0 con 0x7fdd340b4d60 2024-03-25T19:34:48.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.754+0000 7fdd317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:48.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.754+0000 7fdd317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:48.754 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:34:48.754 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:48.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.756+0000 7fdd3ab0e640 10 librados: watch_flush enter 2024-03-25T19:34:48.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.756+0000 7fdd3ab0e640 10 librados: watch_flush exit 2024-03-25T19:34:48.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.756+0000 7fdd3ab0e640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T19:34:48.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.756+0000 7fdd3ab0e640 10 client.4483.objecter successfully canceled tick 2024-03-25T19:34:48.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.757+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/1025843622 >> v1:172.21.15.188:6800/2207506229 conn(0x7fdd1806dfb0 legacy=0x7fdd18070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.757+0000 7fdd3ab0e640 10 monclient: shutdown 2024-03-25T19:34:48.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.757+0000 7fdd3ab0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:48.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.757+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/1025843622 >> v1:172.21.15.188:6789/0 conn(0x7fdd340b4d60 legacy=0x7fdd34143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.757+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/1025843622 shutdown_connections 2024-03-25T19:34:48.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.757+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/1025843622 >> 172.21.15.19:0/1025843622 conn(0x7fdd340617a0 msgr2=0x7fdd340b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:48.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.757+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/1025843622 shutdown_connections 2024-03-25T19:34:48.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.757+0000 7fdd3ab0e640 1 -- 172.21.15.19:0/1025843622 wait complete. 2024-03-25T19:34:48.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:48.757+0000 7fdd3ab0e640 1 librados: shutdown 2024-03-25T19:34:48.770 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:52.690 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:15.419 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.742 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:15.471 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.777 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:15.507 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.785 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:06:15.488 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.798 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:06:15.501 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.809 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:15.539 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.819 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:15.548 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.820 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:06:15.523 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.846 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:15.576 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:55.577 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:55.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.694+0000 7f4543f67640 10 monclient: get_monmap_and_config 2024-03-25T19:34:55.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.694+0000 7f4543f67640 10 monclient: build_initial_monmap 2024-03-25T19:34:55.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.694+0000 7f4543f67640 10 monclient: monmap: 2024-03-25T19:34:55.695 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:55.695 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:55.695 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:34:55.695894+0000 2024-03-25T19:34:55.695 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:34:55.695894+0000 2024-03-25T19:34:55.695 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:55.695 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:55.695 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:55.695 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:34:55.695 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:34:55.695 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:55.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.695+0000 7f4543f67640 1 Processor -- start 2024-03-25T19:34:55.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.695+0000 7f4543f67640 1 -- start start 2024-03-25T19:34:55.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.695+0000 7f4543f67640 10 monclient: init 2024-03-25T19:34:55.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.696+0000 7f4543f67640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:55.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.696+0000 7f4543f67640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:55.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.696+0000 7f4543f67640 10 monclient(hunting): picked mon.noname-a con 0x7f453c0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:55.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.696+0000 7f4543f67640 10 monclient(hunting): picked mon.noname-c con 0x7f453c13f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:55.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.696+0000 7f4543f67640 10 monclient(hunting): picked mon.noname-b con 0x7f453c1430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:55.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.696+0000 7f4543f67640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c142520 con 0x7f453c0a98f0 2024-03-25T19:34:55.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.696+0000 7f4543f67640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c0b28a0 con 0x7f453c13f710 2024-03-25T19:34:55.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.696+0000 7f4543f67640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c0b2a30 con 0x7f453c1430c0 2024-03-25T19:34:55.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.696+0000 7f4543f67640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:55.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.696+0000 7f4543f67640 10 monclient(hunting): authenticate will time out at 6348.966174s 2024-03-25T19:34:55.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.700+0000 7f4542504640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f453c1430c0 0x7f453c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:42924/0 (socket says 172.21.15.19:42924) 2024-03-25T19:34:55.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.700+0000 7f4542504640 1 -- 172.21.15.19:0/349702845 learned_addr learned my addr 172.21.15.19:0/349702845 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:55.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.707+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827949554 0 0) 0x7f453c0b2a30 con 0x7f453c1430c0 2024-03-25T19:34:55.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.707+0000 7f4540d01640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.707+0000 7f4540d01640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.707+0000 7f4540d01640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.707+0000 7f4540d01640 10 monclient(hunting): my global_id is 4466 2024-03-25T19:34:55.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.707+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4528003520 con 0x7f453c1430c0 2024-03-25T19:34:55.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.708+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571373061 0 0) 0x7f453c0b28a0 con 0x7f453c13f710 2024-03-25T19:34:55.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.708+0000 7f4540d01640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.708+0000 7f4540d01640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.708+0000 7f4540d01640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.708+0000 7f4540d01640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:34:55.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.708+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453c0b2a30 con 0x7f453c13f710 2024-03-25T19:34:55.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.716+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203960647 0 0) 0x7f453c142520 con 0x7f453c0a98f0 2024-03-25T19:34:55.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.716+0000 7f4540d01640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.716+0000 7f4540d01640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.716+0000 7f4540d01640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.716+0000 7f4540d01640 10 monclient(hunting): my global_id is 4494 2024-03-25T19:34:55.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.716+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453c0b28a0 con 0x7f453c0a98f0 2024-03-25T19:34:55.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1788165780 0 0) 0x7f453c0b2a30 con 0x7f453c13f710 2024-03-25T19:34:55.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 >> v1:172.21.15.19:6790/0 conn(0x7f453c1430c0 legacy=0x7f453c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 >> v1:172.21.15.19:6789/0 conn(0x7f453c0a98f0 legacy=0x7f453c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 10 monclient: _finish_hunting 0 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 1 monclient: found mon.noname-c 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c0b44c0 con 0x7f453c13f710 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 10 monclient: _finish_auth 0 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 10 monclient: _check_auth_tickets 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f452c001670 con 0x7f453c13f710 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:55.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4540d01640 10 monclient: dump: 2024-03-25T19:34:55.730 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:55.730 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:55.730 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:55.730 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:55.730 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.730 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:55.730 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:55.730 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:55.730 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:55.730 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:55.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4543f67640 5 monclient: authenticate success, global_id 4486 2024-03-25T19:34:55.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.728+0000 7f4543f67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:55.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.739+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f452c0017d0 con 0x7f453c13f710 2024-03-25T19:34:55.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.739+0000 7f4540d01640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.739+0000 7f4540d01640 1 -- 172.21.15.19:0/349702845 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f452c001be0 con 0x7f453c13f710 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.739+0000 7f4543f67640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.739+0000 7f4543f67640 4 set_mon_vals no callback set 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.739+0000 7f4540d01640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.739+0000 7f4540d01640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.739+0000 7f4540d01640 10 monclient: dump: 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:55.741 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:55.741 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.740+0000 7f4543f67640 10 monclient: shutdown 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.740+0000 7f4543f67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.740+0000 7f4543f67640 1 -- 172.21.15.19:0/349702845 >> v1:172.21.15.188:6789/0 conn(0x7f453c13f710 legacy=0x7f453c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.740+0000 7f4543f67640 1 -- 172.21.15.19:0/349702845 shutdown_connections 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.740+0000 7f4543f67640 1 -- 172.21.15.19:0/349702845 >> 172.21.15.19:0/349702845 conn(0x7f453c0b7ea0 msgr2=0x7f453c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.740+0000 7f4543f67640 1 -- 172.21.15.19:0/349702845 shutdown_connections 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.740+0000 7f4543f67640 1 -- 172.21.15.19:0/349702845 wait complete. 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.741+0000 7f4543f67640 10 monclient: build_initial_monmap 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.741+0000 7f4543f67640 10 monclient: monmap: 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:34:55.743 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.741+0000 7f4543f67640 1 librados: starting msgr at 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.741+0000 7f4543f67640 1 librados: starting objecter 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.741+0000 7f4543f67640 1 Processor -- start 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.741+0000 7f4543f67640 1 -- start start 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.741+0000 7f4543f67640 1 librados: setting wanted keys 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.741+0000 7f4543f67640 1 librados: calling monclient init 2024-03-25T19:34:55.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.741+0000 7f4543f67640 10 monclient: init 2024-03-25T19:34:55.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.742+0000 7f4543f67640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:55.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.742+0000 7f4543f67640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:55.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.742+0000 7f4543f67640 10 monclient(hunting): picked mon.noname-a con 0x7f453c0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:34:55.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.742+0000 7f4543f67640 10 monclient(hunting): picked mon.noname-b con 0x7f453c13f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:34:55.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.742+0000 7f4543f67640 10 monclient(hunting): picked mon.noname-c con 0x7f453c1430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:34:55.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.742+0000 7f4543f67640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c1d7490 con 0x7f453c0a98f0 2024-03-25T19:34:55.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.742+0000 7f4543f67640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c0abb90 con 0x7f453c13f710 2024-03-25T19:34:55.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.742+0000 7f4543f67640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c1d76c0 con 0x7f453c1430c0 2024-03-25T19:34:55.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.742+0000 7f4543f67640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:55.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.742+0000 7f4543f67640 10 monclient(hunting): authenticate will time out at 6349.012566s 2024-03-25T19:34:55.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.750+0000 7f4542504640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f453c1430c0 0x7f453c133f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:42952/0 (socket says 172.21.15.19:42952) 2024-03-25T19:34:55.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.750+0000 7f4542504640 1 -- 172.21.15.19:0/2294150053 learned_addr learned my addr 172.21.15.19:0/2294150053 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:34:55.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.752+0000 7f45327fc640 10 client.?.objecter ms_handle_connect 0x7f453c0a98f0 2024-03-25T19:34:55.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.752+0000 7f45327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:55.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.752+0000 7f45327fc640 10 client.?.objecter ms_handle_connect 0x7f453c13f710 2024-03-25T19:34:55.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.752+0000 7f45327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:55.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.753+0000 7f45327fc640 10 client.?.objecter ms_handle_connect 0x7f453c1430c0 2024-03-25T19:34:55.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.753+0000 7f45327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:55.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.759+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401226632 0 0) 0x7f453c0abb90 con 0x7f453c13f710 2024-03-25T19:34:55.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.759+0000 7f45327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.759+0000 7f45327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.759+0000 7f45327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.759+0000 7f45327fc640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:34:55.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.759+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4520004eb0 con 0x7f453c13f710 2024-03-25T19:34:55.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.762+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988799254 0 0) 0x7f453c1d76c0 con 0x7f453c1430c0 2024-03-25T19:34:55.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.762+0000 7f45327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.762+0000 7f45327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.762+0000 7f45327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.762+0000 7f45327fc640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:34:55.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.762+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453c0abb90 con 0x7f453c1430c0 2024-03-25T19:34:55.769 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.769+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478135222 0 0) 0x7f453c1d7490 con 0x7f453c0a98f0 2024-03-25T19:34:55.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.769+0000 7f45327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.769+0000 7f45327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.769+0000 7f45327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.770+0000 7f45327fc640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:34:55.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.770+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453c1d76c0 con 0x7f453c0a98f0 2024-03-25T19:34:55.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.782+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2569905905 0 0) 0x7f4520004eb0 con 0x7f453c13f710 2024-03-25T19:34:55.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.782+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f453c1d7490 con 0x7f453c13f710 2024-03-25T19:34:55.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.782+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f452c001ba0 con 0x7f453c13f710 2024-03-25T19:34:55.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.782+0000 7f45327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.782+0000 7f45327fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.782+0000 7f45327fc640 10 monclient(hunting): dump: 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:55.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.782+0000 7f45327fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:55.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.785+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2491385639 0 0) 0x7f453c0abb90 con 0x7f453c1430c0 2024-03-25T19:34:55.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.785+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4520004eb0 con 0x7f453c1430c0 2024-03-25T19:34:55.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.785+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f4534001b30 con 0x7f453c1430c0 2024-03-25T19:34:55.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.785+0000 7f45327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.785+0000 7f45327fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:55.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.785+0000 7f45327fc640 10 monclient(hunting): dump: 2024-03-25T19:34:55.785 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:55.785 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:55.785 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:55.785 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:55.785 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.786 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:55.786 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:55.786 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:55.786 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:55.786 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:55.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.792+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2103099220 0 0) 0x7f453c1d76c0 con 0x7f453c0a98f0 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.793+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f453c0abb90 con 0x7f453c0a98f0 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.793+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f4538001830 con 0x7f453c0a98f0 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.793+0000 7f45327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.793+0000 7f45327fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.793+0000 7f45327fc640 10 monclient(hunting): dump: 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:55.793 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:55.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1533222027 0 0) 0x7f453c0abb90 con 0x7f453c0a98f0 2024-03-25T19:34:55.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 >> v1:172.21.15.19:6790/0 conn(0x7f453c1430c0 legacy=0x7f453c133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 >> v1:172.21.15.188:6789/0 conn(0x7f453c13f710 legacy=0x7f453c138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f45327fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f45327fc640 1 monclient: found mon.a 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f45327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f45327fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c1d82f0 con 0x7f453c0a98f0 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f45327fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f45327fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f45327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f4543f67640 5 monclient: authenticate success, global_id 4497 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f4543f67640 10 monclient: _renew_subs 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f4543f67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f4543f67640 1 -- 172.21.15.19:0/2294150053 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453c138c20 con 0x7f453c0a98f0 2024-03-25T19:34:55.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f4543f67640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:55.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f4543f67640 10 monclient: _renew_subs 2024-03-25T19:34:55.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f4543f67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:34:55.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f4543f67640 1 -- 172.21.15.19:0/2294150053 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f453c1379e0 con 0x7f453c0a98f0 2024-03-25T19:34:55.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.806+0000 7f4543f67640 1 librados: init done 2024-03-25T19:34:55.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.807+0000 7f4543f67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:55.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.807+0000 7f4543f67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:55.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.807+0000 7f4543f67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:34:55.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:55.807+0000 7f4543f67640 1 -- 172.21.15.19:0/2294150053 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f453c0b3ef0 con 0x7f453c0a98f0 2024-03-25T19:34:56.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.121+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4538001d40 con 0x7f453c0a98f0 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.121+0000 7f45327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.121+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f45380037b0 con 0x7f453c0a98f0 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.121+0000 7f45327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.121+0000 7f45327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.121+0000 7f45327fc640 10 monclient: dump: 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:34:56.122 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:34:56.123 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:34:56.123 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:34:56.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.121+0000 7f4533fff640 4 set_mon_vals no callback set 2024-03-25T19:34:56.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.122+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f4538019840 con 0x7f453c0a98f0 2024-03-25T19:34:56.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.123+0000 7f4541502640 10 monclient: get_auth_request con 0x7f452006e160 auth_method 0 2024-03-25T19:34:56.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.123+0000 7f45327fc640 10 client.4497.objecter ms_handle_connect 0x7f452006e160 2024-03-25T19:34:56.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.128+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f4538053500 con 0x7f453c0a98f0 2024-03-25T19:34:56.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.128+0000 7f45327fc640 10 client.4497.objecter ms_dispatch 0x7f453c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:56.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.128+0000 7f45327fc640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:56.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.128+0000 7f45327fc640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:56.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.128+0000 7f45327fc640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T19:34:56.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.128+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f453804ee40 con 0x7f453c0a98f0 2024-03-25T19:34:56.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.128+0000 7f45327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:56.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.128+0000 7f45327fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:56.349 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.349+0000 7f4543f67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:56.349 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.349+0000 7f4543f67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:56.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.349+0000 7f4543f67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:34:56.350 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.349+0000 7f4543f67640 1 -- 172.21.15.19:0/2294150053 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f453c14f320 con 0x7f453c0a98f0 2024-03-25T19:34:56.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.385+0000 7f45327fc640 1 -- 172.21.15.19:0/2294150053 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f4538050f40 con 0x7f453c0a98f0 2024-03-25T19:34:56.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.385+0000 7f45327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:56.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.385+0000 7f45327fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:56.385 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:34:56.385 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:56.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 10 librados: watch_flush enter 2024-03-25T19:34:56.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 10 librados: watch_flush exit 2024-03-25T19:34:56.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T19:34:56.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 10 client.4497.objecter successfully canceled tick 2024-03-25T19:34:56.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 1 -- 172.21.15.19:0/2294150053 >> v1:172.21.15.188:6800/2207506229 conn(0x7f452006e160 legacy=0x7f4520070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:56.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 10 monclient: shutdown 2024-03-25T19:34:56.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:56.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 1 -- 172.21.15.19:0/2294150053 >> v1:172.21.15.19:6789/0 conn(0x7f453c0a98f0 legacy=0x7f453c1d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:56.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 1 -- 172.21.15.19:0/2294150053 shutdown_connections 2024-03-25T19:34:56.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 1 -- 172.21.15.19:0/2294150053 >> 172.21.15.19:0/2294150053 conn(0x7f453c0b7ea0 msgr2=0x7f453c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:56.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 1 -- 172.21.15.19:0/2294150053 shutdown_connections 2024-03-25T19:34:56.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 1 -- 172.21.15.19:0/2294150053 wait complete. 2024-03-25T19:34:56.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:34:56.388+0000 7f4543f67640 1 librados: shutdown 2024-03-25T19:34:56.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:03.207 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:03.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.323+0000 7f06101a8640 10 monclient: get_monmap_and_config 2024-03-25T19:35:03.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.323+0000 7f06101a8640 10 monclient: build_initial_monmap 2024-03-25T19:35:03.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.323+0000 7f06101a8640 10 monclient: monmap: 2024-03-25T19:35:03.324 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:03.324 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:03.324 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:35:03.324990+0000 2024-03-25T19:35:03.324 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:35:03.324990+0000 2024-03-25T19:35:03.324 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:03.324 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:03.324 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:03.324 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:35:03.324 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:35:03.324 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:03.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.324+0000 7f06101a8640 1 Processor -- start 2024-03-25T19:35:03.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.324+0000 7f06101a8640 1 -- start start 2024-03-25T19:35:03.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.324+0000 7f06101a8640 10 monclient: init 2024-03-25T19:35:03.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.325+0000 7f06101a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:03.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.325+0000 7f06101a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:03.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.325+0000 7f06101a8640 10 monclient(hunting): picked mon.noname-c con 0x7f06080a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:03.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.325+0000 7f06101a8640 10 monclient(hunting): picked mon.noname-a con 0x7f060813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:03.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.325+0000 7f06101a8640 10 monclient(hunting): picked mon.noname-b con 0x7f06081430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:03.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.325+0000 7f06101a8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0608142520 con 0x7f060813f710 2024-03-25T19:35:03.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.325+0000 7f06101a8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06080b28a0 con 0x7f06080a98f0 2024-03-25T19:35:03.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.325+0000 7f06101a8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06080b2a30 con 0x7f06081430c0 2024-03-25T19:35:03.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.325+0000 7f06101a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:03.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.325+0000 7f06101a8640 10 monclient(hunting): authenticate will time out at 6356.595348s 2024-03-25T19:35:03.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.332+0000 7f060df44640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f06080a98f0 0x7f06080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:48156/0 (socket says 172.21.15.19:48156) 2024-03-25T19:35:03.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.332+0000 7f060df44640 1 -- 172.21.15.19:0/824757730 learned_addr learned my addr 172.21.15.19:0/824757730 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:03.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.346+0000 7f060cf42640 1 -- 172.21.15.19:0/824757730 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160667897 0 0) 0x7f06080b28a0 con 0x7f06080a98f0 2024-03-25T19:35:03.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.346+0000 7f060cf42640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.346+0000 7f060cf42640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.346+0000 7f060cf42640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.346+0000 7f060cf42640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:35:03.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.346+0000 7f060cf42640 1 -- 172.21.15.19:0/824757730 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05f4003520 con 0x7f06080a98f0 2024-03-25T19:35:03.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.362+0000 7f060cf42640 1 -- 172.21.15.19:0/824757730 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579567923 0 0) 0x7f06080b2a30 con 0x7f06081430c0 2024-03-25T19:35:03.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.362+0000 7f060cf42640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.362+0000 7f060cf42640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.362+0000 7f060cf42640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.362+0000 7f060cf42640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:35:03.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.362+0000 7f060cf42640 1 -- 172.21.15.19:0/824757730 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06080b28a0 con 0x7f06081430c0 2024-03-25T19:35:03.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 1 -- 172.21.15.19:0/824757730 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1988428372 0 0) 0x7f05f4003520 con 0x7f06080a98f0 2024-03-25T19:35:03.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 1 -- 172.21.15.19:0/824757730 >> v1:172.21.15.19:6790/0 conn(0x7f06081430c0 legacy=0x7f060814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 1 -- 172.21.15.19:0/824757730 >> v1:172.21.15.19:6789/0 conn(0x7f060813f710 legacy=0x7f06081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 10 monclient: _finish_hunting 0 2024-03-25T19:35:03.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 1 monclient: found mon.noname-c 2024-03-25T19:35:03.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:03.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:35:03.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 1 -- 172.21.15.19:0/824757730 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06080b44c0 con 0x7f06080a98f0 2024-03-25T19:35:03.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 10 monclient: _finish_auth 0 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 10 monclient: _check_auth_tickets 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 1 -- 172.21.15.19:0/824757730 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0604001690 con 0x7f06080a98f0 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f06101a8640 5 monclient: authenticate success, global_id 4492 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f060cf42640 10 monclient: dump: 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:03.381 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:03.382 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:03.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.380+0000 7f06101a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.408+0000 7f060cf42640 1 -- 172.21.15.19:0/824757730 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06040017f0 con 0x7f06080a98f0 2024-03-25T19:35:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.408+0000 7f060cf42640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.408+0000 7f060cf42640 1 -- 172.21.15.19:0/824757730 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0604001c00 con 0x7f06080a98f0 2024-03-25T19:35:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.408+0000 7f06101a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.408+0000 7f06101a8640 4 set_mon_vals no callback set 2024-03-25T19:35:03.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.408+0000 7f060cf42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.408+0000 7f060cf42640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:03.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.408+0000 7f060cf42640 10 monclient: dump: 2024-03-25T19:35:03.409 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:03.409 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:03.409 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:03.409 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:03.409 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.409 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:03.409 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:03.409 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:03.409 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:03.409 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:03.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.409+0000 7f06101a8640 10 monclient: shutdown 2024-03-25T19:35:03.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.409+0000 7f06101a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:03.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.409+0000 7f06101a8640 1 -- 172.21.15.19:0/824757730 >> v1:172.21.15.188:6789/0 conn(0x7f06080a98f0 legacy=0x7f06080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.409+0000 7f06101a8640 1 -- 172.21.15.19:0/824757730 shutdown_connections 2024-03-25T19:35:03.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.409+0000 7f06101a8640 1 -- 172.21.15.19:0/824757730 >> 172.21.15.19:0/824757730 conn(0x7f06080b7ea0 msgr2=0x7f06080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:03.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.409+0000 7f06101a8640 1 -- 172.21.15.19:0/824757730 shutdown_connections 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.409+0000 7f06101a8640 1 -- 172.21.15.19:0/824757730 wait complete. 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.410+0000 7f06101a8640 10 monclient: build_initial_monmap 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.410+0000 7f06101a8640 10 monclient: monmap: 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.410+0000 7f06101a8640 1 librados: starting msgr at 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.410+0000 7f06101a8640 1 librados: starting objecter 2024-03-25T19:35:03.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.410+0000 7f06101a8640 1 Processor -- start 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.410+0000 7f06101a8640 1 -- start start 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.410+0000 7f06101a8640 1 librados: setting wanted keys 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.410+0000 7f06101a8640 1 librados: calling monclient init 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.410+0000 7f06101a8640 10 monclient: init 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.411+0000 7f06101a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.411+0000 7f06101a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.411+0000 7f06101a8640 10 monclient(hunting): picked mon.noname-b con 0x7f06080a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.411+0000 7f06101a8640 10 monclient(hunting): picked mon.noname-a con 0x7f060813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.411+0000 7f06101a8640 10 monclient(hunting): picked mon.noname-c con 0x7f06081430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.411+0000 7f06101a8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06081dac50 con 0x7f060813f710 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.411+0000 7f06101a8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06080b16c0 con 0x7f06080a98f0 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.411+0000 7f06101a8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0608136b70 con 0x7f06081430c0 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.411+0000 7f06101a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:03.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.411+0000 7f06101a8640 10 monclient(hunting): authenticate will time out at 6356.681236s 2024-03-25T19:35:03.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.414+0000 7f060d743640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f060813f710 0x7f06081db0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:33882/0 (socket says 172.21.15.19:33882) 2024-03-25T19:35:03.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.414+0000 7f060d743640 1 -- 172.21.15.19:0/1569852881 learned_addr learned my addr 172.21.15.19:0/1569852881 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:03.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.416+0000 7f05fe7fc640 10 client.?.objecter ms_handle_connect 0x7f060813f710 2024-03-25T19:35:03.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.416+0000 7f05fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:03.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.426+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988966647 0 0) 0x7f06081dac50 con 0x7f060813f710 2024-03-25T19:35:03.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.426+0000 7f05fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.426+0000 7f05fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.426+0000 7f05fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.426+0000 7f05fe7fc640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:35:03.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.426+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05ec004eb0 con 0x7f060813f710 2024-03-25T19:35:03.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.436+0000 7f05fe7fc640 10 client.?.objecter ms_handle_connect 0x7f06080a98f0 2024-03-25T19:35:03.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.436+0000 7f05fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:03.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.446+0000 7f05fe7fc640 10 client.?.objecter ms_handle_connect 0x7f06081430c0 2024-03-25T19:35:03.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.446+0000 7f05fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:03.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.452+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229833652 0 0) 0x7f06080b16c0 con 0x7f06080a98f0 2024-03-25T19:35:03.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.452+0000 7f05fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.452+0000 7f05fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.452+0000 7f05fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.452+0000 7f05fe7fc640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:35:03.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.452+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06081dac50 con 0x7f06080a98f0 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.453+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3376579772 0 0) 0x7f05ec004eb0 con 0x7f060813f710 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.453+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06080b16c0 con 0x7f060813f710 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.453+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f05f8001a30 con 0x7f060813f710 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.453+0000 7f05fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.453+0000 7f05fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.453+0000 7f05fe7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:03.454 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:03.455 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:03.455 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:03.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.453+0000 7f05fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:03.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.460+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413224783 0 0) 0x7f0608136b70 con 0x7f06081430c0 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.460+0000 7f05fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.460+0000 7f05fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.460+0000 7f05fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.460+0000 7f05fe7fc640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.460+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05ec004eb0 con 0x7f06081430c0 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.460+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3816720455 0 0) 0x7f06081dac50 con 0x7f06080a98f0 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.460+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0608136b70 con 0x7f06080a98f0 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.460+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0604001b00 con 0x7f06080a98f0 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.460+0000 7f05fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.460+0000 7f05fe7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.460+0000 7f05fe7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:03.462 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 366453400 0 0) 0x7f06080b16c0 con 0x7f060813f710 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 >> v1:172.21.15.19:6790/0 conn(0x7f06081430c0 legacy=0x7f06081e2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 >> v1:172.21.15.188:6789/0 conn(0x7f06080a98f0 legacy=0x7f06081375b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f05fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f05fe7fc640 1 monclient: found mon.a 2024-03-25T19:35:03.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f05fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f05fe7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06080b11a0 con 0x7f060813f710 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f05fe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f05fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f05fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f06101a8640 5 monclient: authenticate success, global_id 4503 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f06101a8640 10 monclient: _renew_subs 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f06101a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f06101a8640 1 -- 172.21.15.19:0/1569852881 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06080b13f0 con 0x7f060813f710 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f06101a8640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f06101a8640 10 monclient: _renew_subs 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f06101a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f06101a8640 1 -- 172.21.15.19:0/1569852881 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0608137ea0 con 0x7f060813f710 2024-03-25T19:35:03.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.461+0000 7f06101a8640 1 librados: init done 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.462+0000 7f06101a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.462+0000 7f06101a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.462+0000 7f06101a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:03.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.462+0000 7f06101a8640 1 -- 172.21.15.19:0/1569852881 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06080627b0 con 0x7f060813f710 2024-03-25T19:35:03.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.681+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05f8001c70 con 0x7f060813f710 2024-03-25T19:35:03.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.681+0000 7f05fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:03.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.681+0000 7f05fffff640 4 set_mon_vals no callback set 2024-03-25T19:35:03.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.778+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f05f8003a60 con 0x7f060813f710 2024-03-25T19:35:03.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.778+0000 7f05fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.778+0000 7f05fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:03.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.778+0000 7f05fe7fc640 10 monclient: dump: 2024-03-25T19:35:03.778 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:03.778 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:03.778 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:03.778 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:03.778 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.778 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:03.778 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:03.778 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:03.778 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:03.779 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:03.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.778+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f05f8003c40 con 0x7f060813f710 2024-03-25T19:35:03.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.779+0000 7f060df44640 10 monclient: get_auth_request con 0x7f05ec06df70 auth_method 0 2024-03-25T19:35:03.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.779+0000 7f05fe7fc640 10 client.4503.objecter ms_handle_connect 0x7f05ec06df70 2024-03-25T19:35:03.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.786+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f05f8054950 con 0x7f060813f710 2024-03-25T19:35:03.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.786+0000 7f05fe7fc640 10 client.4503.objecter ms_dispatch 0x7f0608000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:03.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.786+0000 7f05fe7fc640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:03.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.786+0000 7f05fe7fc640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:03.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.786+0000 7f05fe7fc640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-25T19:35:03.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.786+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f05f8050460 con 0x7f060813f710 2024-03-25T19:35:03.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.786+0000 7f05fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:03.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:03.786+0000 7f05fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:04.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.036+0000 7f06101a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:04.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.036+0000 7f06101a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:04.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.036+0000 7f06101a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:04.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.036+0000 7f06101a8640 1 -- 172.21.15.19:0/1569852881 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f060813e250 con 0x7f060813f710 2024-03-25T19:35:04.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.115+0000 7f05fe7fc640 1 -- 172.21.15.19:0/1569852881 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f05f8052390 con 0x7f060813f710 2024-03-25T19:35:04.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.115+0000 7f05fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:04.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.115+0000 7f05fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:04.115 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:35:04.116 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:04.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.117+0000 7f06101a8640 10 librados: watch_flush enter 2024-03-25T19:35:04.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.118+0000 7f06101a8640 10 librados: watch_flush exit 2024-03-25T19:35:04.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.118+0000 7f06101a8640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-25T19:35:04.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.118+0000 7f06101a8640 10 client.4503.objecter successfully canceled tick 2024-03-25T19:35:04.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.118+0000 7f06101a8640 1 -- 172.21.15.19:0/1569852881 >> v1:172.21.15.188:6800/2207506229 conn(0x7f05ec06df70 legacy=0x7f05ec070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:04.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.118+0000 7f06101a8640 10 monclient: shutdown 2024-03-25T19:35:04.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.118+0000 7f06101a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:04.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.118+0000 7f06101a8640 1 -- 172.21.15.19:0/1569852881 >> v1:172.21.15.19:6789/0 conn(0x7f060813f710 legacy=0x7f06081db0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:04.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.118+0000 7f06101a8640 1 -- 172.21.15.19:0/1569852881 shutdown_connections 2024-03-25T19:35:04.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.118+0000 7f06101a8640 1 -- 172.21.15.19:0/1569852881 >> 172.21.15.19:0/1569852881 conn(0x7f06080b7ea0 msgr2=0x7f0608074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:04.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.118+0000 7f06101a8640 1 -- 172.21.15.19:0/1569852881 shutdown_connections 2024-03-25T19:35:04.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.118+0000 7f06101a8640 1 -- 172.21.15.19:0/1569852881 wait complete. 2024-03-25T19:35:04.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:04.118+0000 7f06101a8640 1 librados: shutdown 2024-03-25T19:35:04.129 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:10.935 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.037+0000 7fb28673a640 10 monclient: get_monmap_and_config 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.037+0000 7fb28673a640 10 monclient: build_initial_monmap 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.037+0000 7fb28673a640 10 monclient: monmap: 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:35:11.037528+0000 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:35:11.037528+0000 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:35:11.036 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:11.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 1 Processor -- start 2024-03-25T19:35:11.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 1 -- start start 2024-03-25T19:35:11.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 10 monclient: init 2024-03-25T19:35:11.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:11.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:11.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 10 monclient(hunting): picked mon.noname-b con 0x7fb2800b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:11.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 10 monclient(hunting): picked mon.noname-a con 0x7fb2800b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:11.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 10 monclient(hunting): picked mon.noname-c con 0x7fb28014be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:11.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2800af940 con 0x7fb2800b1730 2024-03-25T19:35:11.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2800b0a70 con 0x7fb28014be90 2024-03-25T19:35:11.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2800b0d50 con 0x7fb2800b4d60 2024-03-25T19:35:11.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:11.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.038+0000 7fb28673a640 10 monclient(hunting): authenticate will time out at 6364.307672s 2024-03-25T19:35:11.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.048+0000 7fb27f7fe640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fb2800b1730 0x7fb2800b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:44288/0 (socket says 172.21.15.19:44288) 2024-03-25T19:35:11.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.048+0000 7fb27f7fe640 1 -- 172.21.15.19:0/1295516627 learned_addr learned my addr 172.21.15.19:0/1295516627 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.058+0000 7fb27effd640 1 -- 172.21.15.19:0/1295516627 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324245768 0 0) 0x7fb2800af940 con 0x7fb2800b1730 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.058+0000 7fb27effd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.058+0000 7fb27effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.058+0000 7fb27effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.058+0000 7fb27effd640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:35:11.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.058+0000 7fb27effd640 1 -- 172.21.15.19:0/1295516627 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb268003520 con 0x7fb2800b1730 2024-03-25T19:35:11.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 1 -- 172.21.15.19:0/1295516627 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4077215053 0 0) 0x7fb268003520 con 0x7fb2800b1730 2024-03-25T19:35:11.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 1 -- 172.21.15.19:0/1295516627 >> v1:172.21.15.19:6790/0 conn(0x7fb2800b4d60 legacy=0x7fb28013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:35:11.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 1 -- 172.21.15.19:0/1295516627 >> v1:172.21.15.188:6789/0 conn(0x7fb28014be90 legacy=0x7fb28014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:35:11.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:11.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 1 monclient: found mon.noname-a 2024-03-25T19:35:11.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:11.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:35:11.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 1 -- 172.21.15.19:0/1295516627 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2800b1030 con 0x7fb2800b1730 2024-03-25T19:35:11.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 10 monclient: _finish_auth 0 2024-03-25T19:35:11.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:11.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:11.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 1 -- 172.21.15.19:0/1295516627 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb274001690 con 0x7fb2800b1730 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb28673a640 5 monclient: authenticate success, global_id 4506 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb28673a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb27effd640 10 monclient: dump: 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:11.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.066+0000 7fb28673a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:11.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.078+0000 7fb27effd640 1 -- 172.21.15.19:0/1295516627 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb274001860 con 0x7fb2800b1730 2024-03-25T19:35:11.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.078+0000 7fb27effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:11.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.078+0000 7fb27effd640 1 -- 172.21.15.19:0/1295516627 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb274001cb0 con 0x7fb2800b1730 2024-03-25T19:35:11.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.078+0000 7fb27effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.078+0000 7fb27effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:11.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.078+0000 7fb27effd640 10 monclient: dump: 2024-03-25T19:35:11.077 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:11.077 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:11.077 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:11.077 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:11.078 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.078 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:11.078 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:11.078 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:11.078 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:11.078 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:11.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.078+0000 7fb28673a640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:11.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.078+0000 7fb28673a640 4 set_mon_vals no callback set 2024-03-25T19:35:11.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.079+0000 7fb28673a640 10 monclient: shutdown 2024-03-25T19:35:11.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.079+0000 7fb28673a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:11.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.079+0000 7fb28673a640 1 -- 172.21.15.19:0/1295516627 >> v1:172.21.15.19:6789/0 conn(0x7fb2800b1730 legacy=0x7fb2800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.079+0000 7fb28673a640 1 -- 172.21.15.19:0/1295516627 shutdown_connections 2024-03-25T19:35:11.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.079+0000 7fb28673a640 1 -- 172.21.15.19:0/1295516627 >> 172.21.15.19:0/1295516627 conn(0x7fb2800617a0 msgr2=0x7fb280061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.079+0000 7fb28673a640 1 -- 172.21.15.19:0/1295516627 shutdown_connections 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.079+0000 7fb28673a640 1 -- 172.21.15.19:0/1295516627 wait complete. 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.080+0000 7fb28673a640 10 monclient: build_initial_monmap 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.080+0000 7fb28673a640 10 monclient: monmap: 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.080+0000 7fb28673a640 1 librados: starting msgr at 2024-03-25T19:35:11.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.080+0000 7fb28673a640 1 librados: starting objecter 2024-03-25T19:35:11.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.080+0000 7fb28673a640 1 Processor -- start 2024-03-25T19:35:11.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.080+0000 7fb28673a640 1 -- start start 2024-03-25T19:35:11.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.080+0000 7fb28673a640 1 librados: setting wanted keys 2024-03-25T19:35:11.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.080+0000 7fb28673a640 1 librados: calling monclient init 2024-03-25T19:35:11.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.080+0000 7fb28673a640 10 monclient: init 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.081+0000 7fb28673a640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.081+0000 7fb28673a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.081+0000 7fb28673a640 10 monclient(hunting): picked mon.noname-c con 0x7fb2800b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.081+0000 7fb28673a640 10 monclient(hunting): picked mon.noname-a con 0x7fb2800b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.081+0000 7fb28673a640 10 monclient(hunting): picked mon.noname-b con 0x7fb28014be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.081+0000 7fb28673a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2801d7590 con 0x7fb2800b4d60 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.081+0000 7fb28673a640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28013d0f0 con 0x7fb28014be90 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.081+0000 7fb28673a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28013d280 con 0x7fb2800b1730 2024-03-25T19:35:11.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.081+0000 7fb28673a640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:11.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.081+0000 7fb28673a640 10 monclient(hunting): authenticate will time out at 6364.350254s 2024-03-25T19:35:11.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.085+0000 7fb27f7fe640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fb2800b4d60 0x7fb2801d7ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:44306/0 (socket says 172.21.15.19:44306) 2024-03-25T19:35:11.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.085+0000 7fb27f7fe640 1 -- 172.21.15.19:0/1808496998 learned_addr learned my addr 172.21.15.19:0/1808496998 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:11.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.086+0000 7fb27cff9640 10 client.?.objecter ms_handle_connect 0x7fb2800b4d60 2024-03-25T19:35:11.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.086+0000 7fb27cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:11.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.088+0000 7fb27cff9640 10 client.?.objecter ms_handle_connect 0x7fb28014be90 2024-03-25T19:35:11.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.088+0000 7fb27cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:11.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.099+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231209568 0 0) 0x7fb28013d0f0 con 0x7fb28014be90 2024-03-25T19:35:11.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.099+0000 7fb27cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.099+0000 7fb27cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.099+0000 7fb27cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.099+0000 7fb27cff9640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:35:11.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.099+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb26c004eb0 con 0x7fb28014be90 2024-03-25T19:35:11.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.109+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463424574 0 0) 0x7fb2801d7590 con 0x7fb2800b4d60 2024-03-25T19:35:11.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.109+0000 7fb27cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.109+0000 7fb27cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.109+0000 7fb27cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.109+0000 7fb27cff9640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:35:11.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.109+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb28013d0f0 con 0x7fb2800b4d60 2024-03-25T19:35:11.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.124+0000 7fb27cff9640 10 client.?.objecter ms_handle_connect 0x7fb2800b1730 2024-03-25T19:35:11.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.124+0000 7fb27cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1481689782 0 0) 0x7fb26c004eb0 con 0x7fb28014be90 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2801d7590 con 0x7fb28014be90 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb278001ad0 con 0x7fb28014be90 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 10 monclient(hunting): dump: 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:11.128 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:11.129 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:11.129 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:11.129 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:11.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:11.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2471271008 0 0) 0x7fb28013d0f0 con 0x7fb2800b4d60 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb26c004eb0 con 0x7fb2800b4d60 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb274001b70 con 0x7fb2800b4d60 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.129+0000 7fb27cff9640 10 monclient(hunting): dump: 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:11.130 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:11.131 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:11.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.134+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043208938 0 0) 0x7fb270001090 con 0x7fb2800b1730 2024-03-25T19:35:11.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.134+0000 7fb27cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:11.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.134+0000 7fb27cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:11.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.134+0000 7fb27cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:11.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.134+0000 7fb27cff9640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:35:11.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.134+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb28013d0f0 con 0x7fb2800b1730 2024-03-25T19:35:11.136 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2542360806 0 0) 0x7fb26c004eb0 con 0x7fb2800b4d60 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 >> v1:172.21.15.19:6790/0 conn(0x7fb2800b1730 legacy=0x7fb280132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 >> v1:172.21.15.188:6789/0 conn(0x7fb28014be90 legacy=0x7fb2801397e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb27cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb27cff9640 1 monclient: found mon.a 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb27cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb27cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb280133630 con 0x7fb2800b4d60 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb27cff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb27cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb27cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb28673a640 5 monclient: authenticate success, global_id 4509 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb28673a640 10 monclient: _renew_subs 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb28673a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb28673a640 1 -- 172.21.15.19:0/1808496998 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2801d81c0 con 0x7fb2800b4d60 2024-03-25T19:35:11.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb28673a640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:11.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb28673a640 10 monclient: _renew_subs 2024-03-25T19:35:11.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb28673a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:11.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb28673a640 1 -- 172.21.15.19:0/1808496998 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb28013b9f0 con 0x7fb2800b4d60 2024-03-25T19:35:11.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.137+0000 7fb28673a640 1 librados: init done 2024-03-25T19:35:11.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.138+0000 7fb28673a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:11.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.138+0000 7fb28673a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:11.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.138+0000 7fb28673a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:11.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.138+0000 7fb28673a640 1 -- 172.21.15.19:0/1808496998 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb280133820 con 0x7fb2800b4d60 2024-03-25T19:35:11.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.437+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb274004bf0 con 0x7fb2800b4d60 2024-03-25T19:35:11.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.437+0000 7fb27cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:11.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.437+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fb274004f00 con 0x7fb2800b4d60 2024-03-25T19:35:11.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.437+0000 7fb27cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:11.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.437+0000 7fb27cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:11.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.437+0000 7fb27cff9640 10 monclient: dump: 2024-03-25T19:35:11.436 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:11.436 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:11.436 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:11.436 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:11.437 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:11.437 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:11.437 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:11.437 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:11.437 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:11.437 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:11.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.437+0000 7fb27e7fc640 4 set_mon_vals no callback set 2024-03-25T19:35:11.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.438+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fb27401aed0 con 0x7fb2800b4d60 2024-03-25T19:35:11.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.438+0000 7fb27ffff640 10 monclient: get_auth_request con 0x7fb26c06df50 auth_method 0 2024-03-25T19:35:11.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.438+0000 7fb27cff9640 10 client.4509.objecter ms_handle_connect 0x7fb26c06df50 2024-03-25T19:35:11.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.444+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fb274054940 con 0x7fb2800b4d60 2024-03-25T19:35:11.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.444+0000 7fb27cff9640 10 client.4509.objecter ms_dispatch 0x7fb280000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:11.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.444+0000 7fb27cff9640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:11.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.444+0000 7fb27cff9640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:11.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.444+0000 7fb27cff9640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-25T19:35:11.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.444+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb274050450 con 0x7fb2800b4d60 2024-03-25T19:35:11.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.444+0000 7fb27cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:11.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.444+0000 7fb27cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:11.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.702+0000 7fb28673a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:11.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.702+0000 7fb28673a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:11.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.702+0000 7fb28673a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:11.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.702+0000 7fb28673a640 1 -- 172.21.15.19:0/1808496998 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2800ae6c0 con 0x7fb2800b4d60 2024-03-25T19:35:11.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.739+0000 7fb27cff9640 1 -- 172.21.15.19:0/1808496998 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fb274052380 con 0x7fb2800b4d60 2024-03-25T19:35:11.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.739+0000 7fb27cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:11.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.739+0000 7fb27cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:11.739 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:35:11.739 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:11.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 10 librados: watch_flush enter 2024-03-25T19:35:11.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 10 librados: watch_flush exit 2024-03-25T19:35:11.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-25T19:35:11.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 10 client.4509.objecter successfully canceled tick 2024-03-25T19:35:11.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 1 -- 172.21.15.19:0/1808496998 >> v1:172.21.15.188:6800/2207506229 conn(0x7fb26c06df50 legacy=0x7fb26c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 10 monclient: shutdown 2024-03-25T19:35:11.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:11.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 1 -- 172.21.15.19:0/1808496998 >> v1:172.21.15.19:6789/0 conn(0x7fb2800b4d60 legacy=0x7fb2801d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 1 -- 172.21.15.19:0/1808496998 shutdown_connections 2024-03-25T19:35:11.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 1 -- 172.21.15.19:0/1808496998 >> 172.21.15.19:0/1808496998 conn(0x7fb2800617a0 msgr2=0x7fb2800b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:11.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 1 -- 172.21.15.19:0/1808496998 shutdown_connections 2024-03-25T19:35:11.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 1 -- 172.21.15.19:0/1808496998 wait complete. 2024-03-25T19:35:11.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:11.742+0000 7fb28673a640 1 librados: shutdown 2024-03-25T19:35:11.753 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:12.749 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:35.479 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.816 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:35.545 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.849 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:35.578 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.868 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:35.597 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.876 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:06:35.579 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.880 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:06:35.583 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.881 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:35.611 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.911 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:06:35.613 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.923 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:35.653 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:18.560 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:18.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.682+0000 7f9df1154640 10 monclient: get_monmap_and_config 2024-03-25T19:35:18.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.682+0000 7f9df1154640 10 monclient: build_initial_monmap 2024-03-25T19:35:18.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.682+0000 7f9df1154640 10 monclient: monmap: 2024-03-25T19:35:18.681 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:18.682 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:18.682 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:35:18.682718+0000 2024-03-25T19:35:18.682 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:35:18.682718+0000 2024-03-25T19:35:18.682 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:18.682 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:18.682 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:18.682 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:35:18.682 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:35:18.682 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:18.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 1 Processor -- start 2024-03-25T19:35:18.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 1 -- start start 2024-03-25T19:35:18.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 10 monclient: init 2024-03-25T19:35:18.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:18.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:18.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 10 monclient(hunting): picked mon.noname-a con 0x7f9dec0b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:18.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 10 monclient(hunting): picked mon.noname-c con 0x7f9dec0b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:18.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 10 monclient(hunting): picked mon.noname-b con 0x7f9dec14be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:18.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dec0af940 con 0x7f9dec0b4d60 2024-03-25T19:35:18.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dec0b0a70 con 0x7f9dec0b1730 2024-03-25T19:35:18.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dec0b0d50 con 0x7f9dec14be90 2024-03-25T19:35:18.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:18.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.683+0000 7f9df1154640 10 monclient(hunting): authenticate will time out at 6371.952761s 2024-03-25T19:35:18.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.689+0000 7f9dea59c640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f9dec0b1730 0x7f9dec0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:39050/0 (socket says 172.21.15.19:39050) 2024-03-25T19:35:18.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.689+0000 7f9dea59c640 1 -- 172.21.15.19:0/1383632221 learned_addr learned my addr 172.21.15.19:0/1383632221 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:18.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.699+0000 7f9de9d9b640 1 -- 172.21.15.19:0/1383632221 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640815896 0 0) 0x7f9dec0b0a70 con 0x7f9dec0b1730 2024-03-25T19:35:18.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.699+0000 7f9de9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.699+0000 7f9de9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.699+0000 7f9de9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.699+0000 7f9de9d9b640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:35:18.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.699+0000 7f9de9d9b640 1 -- 172.21.15.19:0/1383632221 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd4003520 con 0x7f9dec0b1730 2024-03-25T19:35:18.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.712+0000 7f9de9d9b640 1 -- 172.21.15.19:0/1383632221 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095724440 0 0) 0x7f9dec0b0d50 con 0x7f9dec14be90 2024-03-25T19:35:18.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.712+0000 7f9de9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.712+0000 7f9de9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.712+0000 7f9de9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.712+0000 7f9de9d9b640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:35:18.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.712+0000 7f9de9d9b640 1 -- 172.21.15.19:0/1383632221 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dec0b0a70 con 0x7f9dec14be90 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 1 -- 172.21.15.19:0/1383632221 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 146513858 0 0) 0x7f9dd4003520 con 0x7f9dec0b1730 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 1 -- 172.21.15.19:0/1383632221 >> v1:172.21.15.19:6790/0 conn(0x7f9dec14be90 legacy=0x7f9dec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 1 -- 172.21.15.19:0/1383632221 >> v1:172.21.15.19:6789/0 conn(0x7f9dec0b4d60 legacy=0x7f9dec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 1 monclient: found mon.noname-c 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 1 -- 172.21.15.19:0/1383632221 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec0b1030 con 0x7f9dec0b1730 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 1 -- 172.21.15.19:0/1383632221 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9de0001690 con 0x7f9dec0b1730 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:18.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9de9d9b640 10 monclient: dump: 2024-03-25T19:35:18.721 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:18.721 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:18.721 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:18.721 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:18.721 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.721 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:18.721 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:18.721 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:18.721 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:18.721 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:18.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9df1154640 5 monclient: authenticate success, global_id 4501 2024-03-25T19:35:18.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.720+0000 7f9df1154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:18.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.738+0000 7f9de9d9b640 1 -- 172.21.15.19:0/1383632221 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9de00017f0 con 0x7f9dec0b1730 2024-03-25T19:35:18.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.738+0000 7f9de9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:18.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.738+0000 7f9de9d9b640 1 -- 172.21.15.19:0/1383632221 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9de0001c00 con 0x7f9dec0b1730 2024-03-25T19:35:18.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.738+0000 7f9df1154640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:18.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.738+0000 7f9df1154640 4 set_mon_vals no callback set 2024-03-25T19:35:18.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.738+0000 7f9de9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.738+0000 7f9de9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:18.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.738+0000 7f9de9d9b640 10 monclient: dump: 2024-03-25T19:35:18.738 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:18.738 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:18.738 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:18.738 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:18.738 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.738 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:18.738 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:18.738 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:18.738 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:18.738 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.739+0000 7f9df1154640 10 monclient: shutdown 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.739+0000 7f9df1154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.739+0000 7f9df1154640 1 -- 172.21.15.19:0/1383632221 >> v1:172.21.15.188:6789/0 conn(0x7f9dec0b1730 legacy=0x7f9dec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.739+0000 7f9df1154640 1 -- 172.21.15.19:0/1383632221 shutdown_connections 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.739+0000 7f9df1154640 1 -- 172.21.15.19:0/1383632221 >> 172.21.15.19:0/1383632221 conn(0x7f9dec0617a0 msgr2=0x7f9dec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.739+0000 7f9df1154640 1 -- 172.21.15.19:0/1383632221 shutdown_connections 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.739+0000 7f9df1154640 1 -- 172.21.15.19:0/1383632221 wait complete. 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.740+0000 7f9df1154640 10 monclient: build_initial_monmap 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.740+0000 7f9df1154640 10 monclient: monmap: 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:35:18.740 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.740+0000 7f9df1154640 1 librados: starting msgr at 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.740+0000 7f9df1154640 1 librados: starting objecter 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.741+0000 7f9df1154640 1 Processor -- start 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.741+0000 7f9df1154640 1 -- start start 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.741+0000 7f9df1154640 1 librados: setting wanted keys 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.741+0000 7f9df1154640 1 librados: calling monclient init 2024-03-25T19:35:18.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.741+0000 7f9df1154640 10 monclient: init 2024-03-25T19:35:18.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.742+0000 7f9df1154640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:18.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.742+0000 7f9df1154640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:18.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.742+0000 7f9df1154640 10 monclient(hunting): picked mon.noname-a con 0x7f9dec0b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:18.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.742+0000 7f9df1154640 10 monclient(hunting): picked mon.noname-b con 0x7f9dec0b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:18.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.742+0000 7f9df1154640 10 monclient(hunting): picked mon.noname-c con 0x7f9dec14be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:18.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.742+0000 7f9df1154640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dec137500 con 0x7f9dec0b1730 2024-03-25T19:35:18.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.742+0000 7f9df1154640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dec1d7940 con 0x7f9dec0b4d60 2024-03-25T19:35:18.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.742+0000 7f9df1154640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dec1d8570 con 0x7f9dec14be90 2024-03-25T19:35:18.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.742+0000 7f9df1154640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:18.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.742+0000 7f9df1154640 10 monclient(hunting): authenticate will time out at 6372.011269s 2024-03-25T19:35:18.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.744+0000 7f9dead9d640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f9dec0b1730 0x7f9dec135d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:48850/0 (socket says 172.21.15.19:48850) 2024-03-25T19:35:18.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.744+0000 7f9dead9d640 1 -- 172.21.15.19:0/4104054974 learned_addr learned my addr 172.21.15.19:0/4104054974 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:18.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.745+0000 7f9dd37fe640 10 client.?.objecter ms_handle_connect 0x7f9dec0b1730 2024-03-25T19:35:18.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.745+0000 7f9dd37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:18.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.753+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284145735 0 0) 0x7f9dec137500 con 0x7f9dec0b1730 2024-03-25T19:35:18.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.753+0000 7f9dd37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.753+0000 7f9dd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.753+0000 7f9dd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.753+0000 7f9dd37fe640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:35:18.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.753+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd8004eb0 con 0x7f9dec0b1730 2024-03-25T19:35:18.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.754+0000 7f9dd37fe640 10 client.?.objecter ms_handle_connect 0x7f9dec14be90 2024-03-25T19:35:18.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.754+0000 7f9dd37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:18.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.761+0000 7f9dd37fe640 10 client.?.objecter ms_handle_connect 0x7f9dec0b4d60 2024-03-25T19:35:18.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.761+0000 7f9dd37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:18.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.764+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 818836260 0 0) 0x7f9dd8004eb0 con 0x7f9dec0b1730 2024-03-25T19:35:18.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.764+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dec137500 con 0x7f9dec0b1730 2024-03-25T19:35:18.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.764+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9ddc001a30 con 0x7f9dec0b1730 2024-03-25T19:35:18.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.764+0000 7f9dd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.764+0000 7f9dd37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:18.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.764+0000 7f9dd37fe640 10 monclient(hunting): dump: 2024-03-25T19:35:18.763 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:18.763 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:18.763 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:18.763 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:18.763 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.764 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:18.764 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:18.764 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:18.764 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:18.764 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:18.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.764+0000 7f9dd37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:18.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.775+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995531942 0 0) 0x7f9dec1d7940 con 0x7f9dec0b4d60 2024-03-25T19:35:18.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.775+0000 7f9dd37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.775+0000 7f9dd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.775+0000 7f9dd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.775+0000 7f9dd37fe640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:35:18.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.775+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dd8004eb0 con 0x7f9dec0b4d60 2024-03-25T19:35:18.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.778+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847795967 0 0) 0x7f9dec1d8570 con 0x7f9dec14be90 2024-03-25T19:35:18.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.778+0000 7f9dd37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.778+0000 7f9dd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.778+0000 7f9dd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.778+0000 7f9dd37fe640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:35:18.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.778+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dec1d7940 con 0x7f9dec14be90 2024-03-25T19:35:18.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1988527850 0 0) 0x7f9dec137500 con 0x7f9dec0b1730 2024-03-25T19:35:18.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 >> v1:172.21.15.19:6790/0 conn(0x7f9dec14be90 legacy=0x7f9dec1380a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 >> v1:172.21.15.188:6789/0 conn(0x7f9dec0b4d60 legacy=0x7f9dec1d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9dd37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:18.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9dd37fe640 1 monclient: found mon.a 2024-03-25T19:35:18.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9dd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:18.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9dd37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:18.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec1d8700 con 0x7f9dec0b1730 2024-03-25T19:35:18.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9dd37fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:18.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9dd37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:18.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9dd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:18.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9df1154640 5 monclient: authenticate success, global_id 4515 2024-03-25T19:35:18.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9df1154640 10 monclient: _renew_subs 2024-03-25T19:35:18.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9df1154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:18.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9df1154640 1 -- 172.21.15.19:0/4104054974 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dec1387b0 con 0x7f9dec0b1730 2024-03-25T19:35:18.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9df1154640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:18.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9df1154640 10 monclient: _renew_subs 2024-03-25T19:35:18.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9df1154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:18.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9df1154640 1 -- 172.21.15.19:0/4104054974 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dec137b00 con 0x7f9dec0b1730 2024-03-25T19:35:18.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.779+0000 7f9df1154640 1 librados: init done 2024-03-25T19:35:18.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.780+0000 7f9df1154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:18.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.780+0000 7f9df1154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:18.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.780+0000 7f9df1154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:18.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:18.780+0000 7f9df1154640 1 -- 172.21.15.19:0/4104054974 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dec0627b0 con 0x7f9dec0b1730 2024-03-25T19:35:19.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.100+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ddc001cb0 con 0x7f9dec0b1730 2024-03-25T19:35:19.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.100+0000 7f9dd37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:19.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.100+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9ddc003a60 con 0x7f9dec0b1730 2024-03-25T19:35:19.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.100+0000 7f9dd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:19.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.100+0000 7f9dd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:19.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.100+0000 7f9dd37fe640 10 monclient: dump: 2024-03-25T19:35:19.099 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:19.100 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:19.100 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:19.100 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:19.100 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:19.100 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:19.100 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:19.100 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:19.100 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:19.100 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:19.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.100+0000 7f9de959a640 4 set_mon_vals no callback set 2024-03-25T19:35:19.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.101+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f9ddc003c80 con 0x7f9dec0b1730 2024-03-25T19:35:19.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.101+0000 7f9dea59c640 10 monclient: get_auth_request con 0x7f9dd806de10 auth_method 0 2024-03-25T19:35:19.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.102+0000 7f9dd37fe640 10 client.4515.objecter ms_handle_connect 0x7f9dd806de10 2024-03-25T19:35:19.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.120+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f9ddc054910 con 0x7f9dec0b1730 2024-03-25T19:35:19.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.121+0000 7f9dd37fe640 10 client.4515.objecter ms_dispatch 0x7f9dec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:19.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.121+0000 7f9dd37fe640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:19.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.121+0000 7f9dd37fe640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:19.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.121+0000 7f9dd37fe640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-25T19:35:19.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.121+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ddc050250 con 0x7f9dec0b1730 2024-03-25T19:35:19.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.121+0000 7f9dd37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:19.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.121+0000 7f9dd37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:19.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.344+0000 7f9df1154640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:19.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.344+0000 7f9df1154640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:19.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.344+0000 7f9df1154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:19.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.344+0000 7f9df1154640 1 -- 172.21.15.19:0/4104054974 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9dec1d7ad0 con 0x7f9dec0b1730 2024-03-25T19:35:19.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.384+0000 7f9dd37fe640 1 -- 172.21.15.19:0/4104054974 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f9ddc052350 con 0x7f9dec0b1730 2024-03-25T19:35:19.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.384+0000 7f9dd37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:19.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.384+0000 7f9dd37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:19.383 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:35:19.383 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.386+0000 7f9df1154640 10 librados: watch_flush enter 2024-03-25T19:35:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.386+0000 7f9df1154640 10 librados: watch_flush exit 2024-03-25T19:35:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.386+0000 7f9df1154640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-25T19:35:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.386+0000 7f9df1154640 10 client.4515.objecter successfully canceled tick 2024-03-25T19:35:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.386+0000 7f9df1154640 1 -- 172.21.15.19:0/4104054974 >> v1:172.21.15.188:6800/2207506229 conn(0x7f9dd806de10 legacy=0x7f9dd80702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.386+0000 7f9df1154640 10 monclient: shutdown 2024-03-25T19:35:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.386+0000 7f9df1154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.386+0000 7f9df1154640 1 -- 172.21.15.19:0/4104054974 >> v1:172.21.15.19:6789/0 conn(0x7f9dec0b1730 legacy=0x7f9dec135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.387+0000 7f9df1154640 1 -- 172.21.15.19:0/4104054974 shutdown_connections 2024-03-25T19:35:19.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.387+0000 7f9df1154640 1 -- 172.21.15.19:0/4104054974 >> 172.21.15.19:0/4104054974 conn(0x7f9dec0617a0 msgr2=0x7f9dec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:19.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.387+0000 7f9df1154640 1 -- 172.21.15.19:0/4104054974 shutdown_connections 2024-03-25T19:35:19.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.387+0000 7f9df1154640 1 -- 172.21.15.19:0/4104054974 wait complete. 2024-03-25T19:35:19.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:19.387+0000 7f9df1154640 1 librados: shutdown 2024-03-25T19:35:19.397 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:26.203 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:26.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.320+0000 7fa8a34ba640 10 monclient: get_monmap_and_config 2024-03-25T19:35:26.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.320+0000 7fa8a34ba640 10 monclient: build_initial_monmap 2024-03-25T19:35:26.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.320+0000 7fa8a34ba640 10 monclient: monmap: 2024-03-25T19:35:26.319 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:26.319 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:26.319 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:35:26.320345+0000 2024-03-25T19:35:26.319 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:35:26.320345+0000 2024-03-25T19:35:26.319 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:26.319 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:26.319 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:26.320 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:35:26.320 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:35:26.320 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:26.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.321+0000 7fa8a34ba640 1 Processor -- start 2024-03-25T19:35:26.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.321+0000 7fa8a34ba640 1 -- start start 2024-03-25T19:35:26.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.321+0000 7fa8a34ba640 10 monclient: init 2024-03-25T19:35:26.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.322+0000 7fa8a34ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:26.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.322+0000 7fa8a34ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:26.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.322+0000 7fa8a34ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa89c0b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:26.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.322+0000 7fa8a34ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa89c0b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:26.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.322+0000 7fa8a34ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa89c14be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:26.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.322+0000 7fa8a34ba640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89c0af940 con 0x7fa89c0b1730 2024-03-25T19:35:26.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.322+0000 7fa8a34ba640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89c0b0a70 con 0x7fa89c0b4d60 2024-03-25T19:35:26.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.322+0000 7fa8a34ba640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89c0b0d50 con 0x7fa89c14be90 2024-03-25T19:35:26.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.322+0000 7fa8a34ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:26.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.322+0000 7fa8a34ba640 10 monclient(hunting): authenticate will time out at 6379.591480s 2024-03-25T19:35:26.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.331+0000 7fa8a0a55640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fa89c0b1730 0x7fa89c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:49480/0 (socket says 172.21.15.19:49480) 2024-03-25T19:35:26.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.331+0000 7fa8a0a55640 1 -- 172.21.15.19:0/4018936803 learned_addr learned my addr 172.21.15.19:0/4018936803 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:26.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.343+0000 7fa893fff640 1 -- 172.21.15.19:0/4018936803 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698750011 0 0) 0x7fa89c0af940 con 0x7fa89c0b1730 2024-03-25T19:35:26.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.344+0000 7fa893fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.344+0000 7fa893fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.344+0000 7fa893fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.344+0000 7fa893fff640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:35:26.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.344+0000 7fa893fff640 1 -- 172.21.15.19:0/4018936803 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa888003520 con 0x7fa89c0b1730 2024-03-25T19:35:26.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 1 -- 172.21.15.19:0/4018936803 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2057684439 0 0) 0x7fa888003520 con 0x7fa89c0b1730 2024-03-25T19:35:26.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 1 -- 172.21.15.19:0/4018936803 >> v1:172.21.15.19:6790/0 conn(0x7fa89c14be90 legacy=0x7fa89c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 1 -- 172.21.15.19:0/4018936803 >> v1:172.21.15.188:6789/0 conn(0x7fa89c0b4d60 legacy=0x7fa89c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:26.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 1 monclient: found mon.noname-a 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 1 -- 172.21.15.19:0/4018936803 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89c0b1030 con 0x7fa89c0b1730 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 10 monclient: _finish_auth 0 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 1 -- 172.21.15.19:0/4018936803 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa898001690 con 0x7fa89c0b1730 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa893fff640 10 monclient: dump: 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.352 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:26.353 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:26.353 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:26.353 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:26.353 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:26.353 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa8a34ba640 5 monclient: authenticate success, global_id 4518 2024-03-25T19:35:26.353 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.352+0000 7fa8a34ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:26.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.361+0000 7fa893fff640 1 -- 172.21.15.19:0/4018936803 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8980017f0 con 0x7fa89c0b1730 2024-03-25T19:35:26.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.361+0000 7fa893fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:26.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.361+0000 7fa893fff640 1 -- 172.21.15.19:0/4018936803 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa898001c40 con 0x7fa89c0b1730 2024-03-25T19:35:26.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.361+0000 7fa8a34ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.361+0000 7fa8a34ba640 4 set_mon_vals no callback set 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.361+0000 7fa893fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.361+0000 7fa893fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.361+0000 7fa893fff640 10 monclient: dump: 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:26.361 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:26.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.362+0000 7fa8a34ba640 10 monclient: shutdown 2024-03-25T19:35:26.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.362+0000 7fa8a34ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:26.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.362+0000 7fa8a34ba640 1 -- 172.21.15.19:0/4018936803 >> v1:172.21.15.19:6789/0 conn(0x7fa89c0b1730 legacy=0x7fa89c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.362+0000 7fa8a34ba640 1 -- 172.21.15.19:0/4018936803 shutdown_connections 2024-03-25T19:35:26.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.362+0000 7fa8a34ba640 1 -- 172.21.15.19:0/4018936803 >> 172.21.15.19:0/4018936803 conn(0x7fa89c0617a0 msgr2=0x7fa89c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:26.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.362+0000 7fa8a34ba640 1 -- 172.21.15.19:0/4018936803 shutdown_connections 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.362+0000 7fa8a34ba640 1 -- 172.21.15.19:0/4018936803 wait complete. 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.363+0000 7fa8a34ba640 10 monclient: build_initial_monmap 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.363+0000 7fa8a34ba640 10 monclient: monmap: 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:26.364 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.363+0000 7fa8a34ba640 1 librados: starting msgr at 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.363+0000 7fa8a34ba640 1 librados: starting objecter 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.363+0000 7fa8a34ba640 1 Processor -- start 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.363+0000 7fa8a34ba640 1 -- start start 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 1 librados: setting wanted keys 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 1 librados: calling monclient init 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 10 monclient: init 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa89c0b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa89c0b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa89c14be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89c134f90 con 0x7fa89c14be90 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89c1369b0 con 0x7fa89c0b1730 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89c136c90 con 0x7fa89c0b4d60 2024-03-25T19:35:26.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:26.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.364+0000 7fa8a34ba640 10 monclient(hunting): authenticate will time out at 6379.633507s 2024-03-25T19:35:26.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.374+0000 7fa8a1256640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fa89c0b1730 0x7fa89c142a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:37606/0 (socket says 172.21.15.19:37606) 2024-03-25T19:35:26.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.374+0000 7fa8a1256640 1 -- 172.21.15.19:0/3914017934 learned_addr learned my addr 172.21.15.19:0/3914017934 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:26.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.375+0000 7fa891ffb640 10 client.?.objecter ms_handle_connect 0x7fa89c0b1730 2024-03-25T19:35:26.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.375+0000 7fa891ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:26.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.382+0000 7fa891ffb640 10 client.?.objecter ms_handle_connect 0x7fa89c0b4d60 2024-03-25T19:35:26.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.382+0000 7fa891ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:26.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.385+0000 7fa891ffb640 10 client.?.objecter ms_handle_connect 0x7fa89c14be90 2024-03-25T19:35:26.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.385+0000 7fa891ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:26.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.387+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398187162 0 0) 0x7fa89c136c90 con 0x7fa89c0b4d60 2024-03-25T19:35:26.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.387+0000 7fa891ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.387+0000 7fa891ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.387+0000 7fa891ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.387+0000 7fa891ffb640 10 monclient(hunting): my global_id is 4487 2024-03-25T19:35:26.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.387+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa880004eb0 con 0x7fa89c0b4d60 2024-03-25T19:35:26.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.390+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242187592 0 0) 0x7fa89c1369b0 con 0x7fa89c0b1730 2024-03-25T19:35:26.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.390+0000 7fa891ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.390+0000 7fa891ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.390+0000 7fa891ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.390+0000 7fa891ffb640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:35:26.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.390+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa89c136c90 con 0x7fa89c0b1730 2024-03-25T19:35:26.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.413+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427459262 0 0) 0x7fa89c134f90 con 0x7fa89c14be90 2024-03-25T19:35:26.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.413+0000 7fa891ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:26.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.413+0000 7fa891ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:26.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.413+0000 7fa891ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:26.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.413+0000 7fa891ffb640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:35:26.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.413+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa89c1369b0 con 0x7fa89c14be90 2024-03-25T19:35:26.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.419+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1641674935 0 0) 0x7fa880004eb0 con 0x7fa89c0b4d60 2024-03-25T19:35:26.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.419+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa89c134f90 con 0x7fa89c0b4d60 2024-03-25T19:35:26.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.419+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa898001b00 con 0x7fa89c0b4d60 2024-03-25T19:35:26.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.419+0000 7fa891ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.419+0000 7fa891ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:26.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.419+0000 7fa891ffb640 10 monclient(hunting): dump: 2024-03-25T19:35:26.419 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:26.419 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:26.419 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:26.419 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:26.419 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.419 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:26.419 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:26.419 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:26.419 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:26.419 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:26.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.419+0000 7fa891ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:26.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.426+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1973297631 0 0) 0x7fa89c136c90 con 0x7fa89c0b1730 2024-03-25T19:35:26.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.426+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa880004eb0 con 0x7fa89c0b1730 2024-03-25T19:35:26.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.426+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa88c001970 con 0x7fa89c0b1730 2024-03-25T19:35:26.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.426+0000 7fa891ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.426+0000 7fa891ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:26.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.426+0000 7fa891ffb640 10 monclient(hunting): dump: 2024-03-25T19:35:26.426 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:26.426 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:26.426 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:26.426 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:26.426 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.426 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:26.426 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:26.426 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:26.426 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:26.426 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:26.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4060544312 0 0) 0x7fa880004eb0 con 0x7fa89c0b1730 2024-03-25T19:35:26.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 >> v1:172.21.15.19:6790/0 conn(0x7fa89c0b4d60 legacy=0x7fa89c134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 >> v1:172.21.15.19:6789/0 conn(0x7fa89c14be90 legacy=0x7fa89c134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa891ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:26.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa891ffb640 1 monclient: found mon.b 2024-03-25T19:35:26.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa891ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:26.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa891ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:35:26.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89c135b00 con 0x7fa89c0b1730 2024-03-25T19:35:26.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa891ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:26.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa891ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:26.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa891ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:26.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa8a34ba640 5 monclient: authenticate success, global_id 4510 2024-03-25T19:35:26.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa8a34ba640 10 monclient: _renew_subs 2024-03-25T19:35:26.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa8a34ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:35:26.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.435+0000 7fa8a34ba640 1 -- 172.21.15.19:0/3914017934 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89c1431a0 con 0x7fa89c0b1730 2024-03-25T19:35:26.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.436+0000 7fa8a34ba640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:26.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.436+0000 7fa8a34ba640 10 monclient: _renew_subs 2024-03-25T19:35:26.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.436+0000 7fa8a34ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:35:26.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.436+0000 7fa8a34ba640 1 -- 172.21.15.19:0/3914017934 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89c142220 con 0x7fa89c0b1730 2024-03-25T19:35:26.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.436+0000 7fa8a34ba640 1 librados: init done 2024-03-25T19:35:26.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.437+0000 7fa8a34ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:26.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.437+0000 7fa8a34ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:26.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.437+0000 7fa8a34ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:35:26.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.437+0000 7fa8a34ba640 1 -- 172.21.15.19:0/3914017934 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89c1424c0 con 0x7fa89c0b1730 2024-03-25T19:35:26.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.714+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa88c003610 con 0x7fa89c0b1730 2024-03-25T19:35:26.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.714+0000 7fa891ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:26.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.714+0000 7fa8937fe640 4 set_mon_vals no callback set 2024-03-25T19:35:26.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.715+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa88c003800 con 0x7fa89c0b1730 2024-03-25T19:35:26.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.715+0000 7fa891ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.715+0000 7fa891ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:26.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.715+0000 7fa891ffb640 10 monclient: dump: 2024-03-25T19:35:26.714 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:26.715 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:26.715 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:26.715 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:26.715 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.715 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:26.715 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:26.715 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:26.715 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:26.715 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:26.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.715+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fa88c003a20 con 0x7fa89c0b1730 2024-03-25T19:35:26.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.716+0000 7fa8a0a55640 10 monclient: get_auth_request con 0x7fa88006dfb0 auth_method 0 2024-03-25T19:35:26.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.716+0000 7fa891ffb640 10 client.4510.objecter ms_handle_connect 0x7fa88006dfb0 2024-03-25T19:35:26.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.719+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fa88c054630 con 0x7fa89c0b1730 2024-03-25T19:35:26.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.719+0000 7fa891ffb640 10 client.4510.objecter ms_dispatch 0x7fa89c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:26.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.719+0000 7fa891ffb640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:26.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.719+0000 7fa891ffb640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:26.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.719+0000 7fa891ffb640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T19:35:26.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.720+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa88c050140 con 0x7fa89c0b1730 2024-03-25T19:35:26.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.720+0000 7fa891ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:26.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.720+0000 7fa891ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:26.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.975+0000 7fa8a34ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:26.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.975+0000 7fa8a34ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:26.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.975+0000 7fa8a34ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:35:26.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:26.975+0000 7fa8a34ba640 1 -- 172.21.15.19:0/3914017934 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa89c135c90 con 0x7fa89c0b1730 2024-03-25T19:35:27.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.103+0000 7fa891ffb640 1 -- 172.21.15.19:0/3914017934 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fa89c135c90 con 0x7fa89c0b1730 2024-03-25T19:35:27.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.103+0000 7fa891ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:27.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.103+0000 7fa891ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:27.103 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:35:27.103 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:27.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.105+0000 7fa8a34ba640 10 librados: watch_flush enter 2024-03-25T19:35:27.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.105+0000 7fa8a34ba640 10 librados: watch_flush exit 2024-03-25T19:35:27.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.105+0000 7fa8a34ba640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T19:35:27.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.105+0000 7fa8a34ba640 10 client.4510.objecter successfully canceled tick 2024-03-25T19:35:27.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.105+0000 7fa8a34ba640 1 -- 172.21.15.19:0/3914017934 >> v1:172.21.15.188:6800/2207506229 conn(0x7fa88006dfb0 legacy=0x7fa880070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:27.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.105+0000 7fa8a34ba640 10 monclient: shutdown 2024-03-25T19:35:27.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.105+0000 7fa8a34ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:27.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.106+0000 7fa8a34ba640 1 -- 172.21.15.19:0/3914017934 >> v1:172.21.15.188:6789/0 conn(0x7fa89c0b1730 legacy=0x7fa89c142a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:27.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.106+0000 7fa8a34ba640 1 -- 172.21.15.19:0/3914017934 shutdown_connections 2024-03-25T19:35:27.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.106+0000 7fa8a34ba640 1 -- 172.21.15.19:0/3914017934 >> 172.21.15.19:0/3914017934 conn(0x7fa89c0617a0 msgr2=0x7fa89c13c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:27.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.106+0000 7fa8a34ba640 1 -- 172.21.15.19:0/3914017934 shutdown_connections 2024-03-25T19:35:27.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.106+0000 7fa8a34ba640 1 -- 172.21.15.19:0/3914017934 wait complete. 2024-03-25T19:35:27.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:27.106+0000 7fa8a34ba640 1 librados: shutdown 2024-03-25T19:35:27.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:32.899 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:55.628 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:32.921 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:55.650 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:32.941 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:06:55.643 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:32.947 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:06:55.650 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:32.951 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:55.680 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:32.973 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:06:55.675 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:32.984 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:55.713 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:32.988 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:55.717 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:33.010 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:55.740 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:33.923 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:34.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.035+0000 7f3bed17b640 10 monclient: get_monmap_and_config 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.035+0000 7f3bed17b640 10 monclient: build_initial_monmap 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.035+0000 7f3bed17b640 10 monclient: monmap: 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:35:34.036606+0000 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:35:34.036606+0000 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:34.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.035+0000 7f3bed17b640 1 Processor -- start 2024-03-25T19:35:34.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.036+0000 7f3bed17b640 1 -- start start 2024-03-25T19:35:34.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.036+0000 7f3bed17b640 10 monclient: init 2024-03-25T19:35:34.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.036+0000 7f3bed17b640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:34.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.036+0000 7f3bed17b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:34.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.036+0000 7f3bed17b640 10 monclient(hunting): picked mon.noname-a con 0x7f3be80b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:34.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.037+0000 7f3bed17b640 10 monclient(hunting): picked mon.noname-b con 0x7f3be80b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:34.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.037+0000 7f3bed17b640 10 monclient(hunting): picked mon.noname-c con 0x7f3be814be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:34.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.037+0000 7f3bed17b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80af940 con 0x7f3be80b4d60 2024-03-25T19:35:34.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.037+0000 7f3bed17b640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80b0a70 con 0x7f3be814be90 2024-03-25T19:35:34.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.037+0000 7f3bed17b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80b0d50 con 0x7f3be80b1730 2024-03-25T19:35:34.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.037+0000 7f3bed17b640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:34.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.037+0000 7f3bed17b640 10 monclient(hunting): authenticate will time out at 6387.307005s 2024-03-25T19:35:34.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.043+0000 7f3be759e640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f3be814be90 0x7f3be814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:44538/0 (socket says 172.21.15.19:44538) 2024-03-25T19:35:34.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.043+0000 7f3be759e640 1 -- 172.21.15.19:0/2429147597 learned_addr learned my addr 172.21.15.19:0/2429147597 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:34.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.055+0000 7f3be5d9b640 1 -- 172.21.15.19:0/2429147597 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161743546 0 0) 0x7f3be80b0d50 con 0x7f3be80b1730 2024-03-25T19:35:34.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.055+0000 7f3be5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:34.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.055+0000 7f3be5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:34.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.055+0000 7f3be5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:34.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.055+0000 7f3be5d9b640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:35:34.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.055+0000 7f3be5d9b640 1 -- 172.21.15.19:0/2429147597 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd0003520 con 0x7f3be80b1730 2024-03-25T19:35:34.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 1 -- 172.21.15.19:0/2429147597 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4263601406 0 0) 0x7f3bd0003520 con 0x7f3be80b1730 2024-03-25T19:35:34.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 1 -- 172.21.15.19:0/2429147597 >> v1:172.21.15.188:6789/0 conn(0x7f3be814be90 legacy=0x7f3be814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 1 -- 172.21.15.19:0/2429147597 >> v1:172.21.15.19:6789/0 conn(0x7f3be80b4d60 legacy=0x7f3be813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 1 monclient: found mon.noname-b 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 1 -- 172.21.15.19:0/2429147597 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be80b1030 con 0x7f3be80b1730 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 10 monclient: _finish_auth 0 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 1 -- 172.21.15.19:0/2429147597 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3bdc001690 con 0x7f3be80b1730 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3be5d9b640 10 monclient: dump: 2024-03-25T19:35:34.064 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:34.065 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:34.065 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:34.065 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:34.065 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:34.065 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:34.065 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:34.065 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:34.065 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:34.065 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:34.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3bed17b640 5 monclient: authenticate success, global_id 4490 2024-03-25T19:35:34.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.063+0000 7f3bed17b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:34.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.080+0000 7f3be5d9b640 1 -- 172.21.15.19:0/2429147597 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bdc001860 con 0x7f3be80b1730 2024-03-25T19:35:34.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.080+0000 7f3be5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:34.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.080+0000 7f3be5d9b640 1 -- 172.21.15.19:0/2429147597 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3bdc001c70 con 0x7f3be80b1730 2024-03-25T19:35:34.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.080+0000 7f3bed17b640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:34.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.080+0000 7f3bed17b640 4 set_mon_vals no callback set 2024-03-25T19:35:34.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.080+0000 7f3be5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:34.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.080+0000 7f3be5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:34.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.080+0000 7f3be5d9b640 10 monclient: dump: 2024-03-25T19:35:34.080 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:34.081 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:34.081 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:34.081 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:34.081 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:34.081 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:34.081 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:34.081 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:34.081 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:34.081 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:34.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.081+0000 7f3bed17b640 10 monclient: shutdown 2024-03-25T19:35:34.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.081+0000 7f3bed17b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:34.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.081+0000 7f3bed17b640 1 -- 172.21.15.19:0/2429147597 >> v1:172.21.15.19:6790/0 conn(0x7f3be80b1730 legacy=0x7f3be80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.081+0000 7f3bed17b640 1 -- 172.21.15.19:0/2429147597 shutdown_connections 2024-03-25T19:35:34.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.081+0000 7f3bed17b640 1 -- 172.21.15.19:0/2429147597 >> 172.21.15.19:0/2429147597 conn(0x7f3be80617a0 msgr2=0x7f3be8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:34.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.081+0000 7f3bed17b640 1 -- 172.21.15.19:0/2429147597 shutdown_connections 2024-03-25T19:35:34.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.081+0000 7f3bed17b640 1 -- 172.21.15.19:0/2429147597 wait complete. 2024-03-25T19:35:34.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.082+0000 7f3bed17b640 10 monclient: build_initial_monmap 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.082+0000 7f3bed17b640 10 monclient: monmap: 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.082+0000 7f3bed17b640 1 librados: starting msgr at 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.082+0000 7f3bed17b640 1 librados: starting objecter 2024-03-25T19:35:34.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.082+0000 7f3bed17b640 1 Processor -- start 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.083+0000 7f3bed17b640 1 -- start start 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.083+0000 7f3bed17b640 1 librados: setting wanted keys 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.083+0000 7f3bed17b640 1 librados: calling monclient init 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.083+0000 7f3bed17b640 10 monclient: init 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.083+0000 7f3bed17b640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.083+0000 7f3bed17b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.084+0000 7f3bed17b640 10 monclient(hunting): picked mon.noname-a con 0x7f3be80b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.084+0000 7f3bed17b640 10 monclient(hunting): picked mon.noname-c con 0x7f3be80b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.084+0000 7f3bed17b640 10 monclient(hunting): picked mon.noname-b con 0x7f3be814be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.084+0000 7f3bed17b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be81450b0 con 0x7f3be80b1730 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.084+0000 7f3bed17b640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be8143b00 con 0x7f3be814be90 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.084+0000 7f3bed17b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be81452e0 con 0x7f3be80b4d60 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.084+0000 7f3bed17b640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:34.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.084+0000 7f3bed17b640 10 monclient(hunting): authenticate will time out at 6387.354463s 2024-03-25T19:35:34.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.085+0000 7f3be759e640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f3be814be90 0x7f3be81db3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:44586/0 (socket says 172.21.15.19:44586) 2024-03-25T19:35:34.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.085+0000 7f3be759e640 1 -- 172.21.15.19:0/1416608538 learned_addr learned my addr 172.21.15.19:0/1416608538 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:34.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.085+0000 7f3bcf7fe640 10 client.?.objecter ms_handle_connect 0x7f3be814be90 2024-03-25T19:35:34.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.085+0000 7f3bcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:34.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.090+0000 7f3bcf7fe640 10 client.?.objecter ms_handle_connect 0x7f3be80b4d60 2024-03-25T19:35:34.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.090+0000 7f3bcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:34.104 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.104+0000 7f3bcf7fe640 10 client.?.objecter ms_handle_connect 0x7f3be80b1730 2024-03-25T19:35:34.104 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.104+0000 7f3bcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:34.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.164+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265182334 0 0) 0x7f3be81450b0 con 0x7f3be80b1730 2024-03-25T19:35:34.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.164+0000 7f3bcf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:34.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.164+0000 7f3bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:34.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.164+0000 7f3bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:34.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.164+0000 7f3bcf7fe640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:35:34.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.164+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd4004eb0 con 0x7f3be80b1730 2024-03-25T19:35:34.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.174+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667817011 0 0) 0x7f3be8143b00 con 0x7f3be814be90 2024-03-25T19:35:34.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.174+0000 7f3bcf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:34.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.174+0000 7f3bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:34.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.174+0000 7f3bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:34.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.174+0000 7f3bcf7fe640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:35:34.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.174+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be81450b0 con 0x7f3be814be90 2024-03-25T19:35:34.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.184+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4272707938 0 0) 0x7f3bd4004eb0 con 0x7f3be80b1730 2024-03-25T19:35:34.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.184+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3be8143b00 con 0x7f3be80b1730 2024-03-25T19:35:34.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.184+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3bd8001870 con 0x7f3be80b1730 2024-03-25T19:35:34.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.184+0000 7f3bcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.184+0000 7f3bcf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.184+0000 7f3bcf7fe640 10 monclient(hunting): dump: 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:34.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.184+0000 7f3bcf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:34.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.194+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 606650473 0 0) 0x7f3be81450b0 con 0x7f3be814be90 2024-03-25T19:35:34.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.194+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bd4004eb0 con 0x7f3be814be90 2024-03-25T19:35:34.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.194+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3be0001b50 con 0x7f3be814be90 2024-03-25T19:35:34.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.194+0000 7f3bcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:34.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.194+0000 7f3bcf7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:34.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.194+0000 7f3bcf7fe640 10 monclient(hunting): dump: 2024-03-25T19:35:34.194 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:34.194 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:34.195 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:34.195 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:34.195 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:34.195 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:34.195 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:34.195 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:34.196 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:34.196 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:34.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 699690453 0 0) 0x7f3be8143b00 con 0x7f3be80b1730 2024-03-25T19:35:34.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 >> v1:172.21.15.19:6790/0 conn(0x7f3be80b4d60 legacy=0x7f3be81dace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 >> v1:172.21.15.188:6789/0 conn(0x7f3be814be90 legacy=0x7f3be81db3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bcf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:34.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bcf7fe640 1 monclient: found mon.a 2024-03-25T19:35:34.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:34.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bcf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:34.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be8143690 con 0x7f3be80b1730 2024-03-25T19:35:34.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bcf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:34.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bcf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:34.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:34.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bed17b640 5 monclient: authenticate success, global_id 4524 2024-03-25T19:35:34.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bed17b640 10 monclient: _renew_subs 2024-03-25T19:35:34.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bed17b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:34.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bed17b640 1 -- 172.21.15.19:0/1416608538 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be8130500 con 0x7f3be80b1730 2024-03-25T19:35:34.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bed17b640 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:34.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bed17b640 10 monclient: _renew_subs 2024-03-25T19:35:34.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bed17b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:34.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bed17b640 1 -- 172.21.15.19:0/1416608538 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be81441a0 con 0x7f3be80b1730 2024-03-25T19:35:34.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.195+0000 7f3bed17b640 1 librados: init done 2024-03-25T19:35:34.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.196+0000 7f3bed17b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:34.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.196+0000 7f3bed17b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:34.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.196+0000 7f3bed17b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:34.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.196+0000 7f3bed17b640 1 -- 172.21.15.19:0/1416608538 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be81438e0 con 0x7f3be80b1730 2024-03-25T19:35:34.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.410+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bd80035f0 con 0x7f3be80b1730 2024-03-25T19:35:34.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.410+0000 7f3bcf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:34.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.410+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3bd8001d80 con 0x7f3be80b1730 2024-03-25T19:35:34.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.410+0000 7f3bcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:34.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.410+0000 7f3bcf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:34.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.410+0000 7f3bcf7fe640 10 monclient: dump: 2024-03-25T19:35:34.410 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:34.410 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:34.411 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:34.411 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:34.411 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:34.411 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:34.411 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:34.411 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:34.411 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:34.411 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:34.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.410+0000 7f3be559a640 4 set_mon_vals no callback set 2024-03-25T19:35:34.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.506+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f3bd8019770 con 0x7f3be80b1730 2024-03-25T19:35:34.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.507+0000 7f3be659c640 10 monclient: get_auth_request con 0x7f3bd406de90 auth_method 0 2024-03-25T19:35:34.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.508+0000 7f3bcf7fe640 10 client.4524.objecter ms_handle_connect 0x7f3bd406de90 2024-03-25T19:35:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.514+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f3bd80534f0 con 0x7f3be80b1730 2024-03-25T19:35:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.514+0000 7f3bcf7fe640 10 client.4524.objecter ms_dispatch 0x7f3be8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.514+0000 7f3bcf7fe640 3 client.4524.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.514+0000 7f3bcf7fe640 3 client.4524.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.515+0000 7f3bcf7fe640 20 client.4524.objecter dump_active .. 0 homeless 2024-03-25T19:35:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.515+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3bd804ee30 con 0x7f3be80b1730 2024-03-25T19:35:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.515+0000 7f3bcf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:34.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.515+0000 7f3bcf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:34.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.784+0000 7f3bed17b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.784+0000 7f3bed17b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.784+0000 7f3bed17b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:34.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.784+0000 7f3bed17b640 1 -- 172.21.15.19:0/1416608538 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3be81401f0 con 0x7f3be80b1730 2024-03-25T19:35:34.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.803+0000 7f3bcf7fe640 1 -- 172.21.15.19:0/1416608538 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f3bd8050f30 con 0x7f3be80b1730 2024-03-25T19:35:34.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.803+0000 7f3bcf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.803+0000 7f3bcf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:34.803 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:35:34.803 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:34.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.805+0000 7f3bed17b640 10 librados: watch_flush enter 2024-03-25T19:35:34.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.805+0000 7f3bed17b640 10 librados: watch_flush exit 2024-03-25T19:35:34.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.805+0000 7f3bed17b640 20 client.4524.objecter shutdown clearing up homeless session... 2024-03-25T19:35:34.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.805+0000 7f3bed17b640 10 client.4524.objecter successfully canceled tick 2024-03-25T19:35:34.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.805+0000 7f3bed17b640 1 -- 172.21.15.19:0/1416608538 >> v1:172.21.15.188:6800/2207506229 conn(0x7f3bd406de90 legacy=0x7f3bd4070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.805+0000 7f3bed17b640 10 monclient: shutdown 2024-03-25T19:35:34.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.805+0000 7f3bed17b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:34.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.805+0000 7f3bed17b640 1 -- 172.21.15.19:0/1416608538 >> v1:172.21.15.19:6789/0 conn(0x7f3be80b1730 legacy=0x7f3be812fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.805+0000 7f3bed17b640 1 -- 172.21.15.19:0/1416608538 shutdown_connections 2024-03-25T19:35:34.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.805+0000 7f3bed17b640 1 -- 172.21.15.19:0/1416608538 >> 172.21.15.19:0/1416608538 conn(0x7f3be80617a0 msgr2=0x7f3be8137f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:34.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.806+0000 7f3bed17b640 1 -- 172.21.15.19:0/1416608538 shutdown_connections 2024-03-25T19:35:34.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.806+0000 7f3bed17b640 1 -- 172.21.15.19:0/1416608538 wait complete. 2024-03-25T19:35:34.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:34.806+0000 7f3bed17b640 1 librados: shutdown 2024-03-25T19:35:34.818 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:41.624 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:41.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.742+0000 7f210e5fe640 10 monclient: get_monmap_and_config 2024-03-25T19:35:41.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.742+0000 7f210e5fe640 10 monclient: build_initial_monmap 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.742+0000 7f210e5fe640 10 monclient: monmap: 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:35:41.743784+0000 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:35:41.743784+0000 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:35:41.743 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:41.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.743+0000 7f210e5fe640 1 Processor -- start 2024-03-25T19:35:41.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.743+0000 7f210e5fe640 1 -- start start 2024-03-25T19:35:41.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.743+0000 7f210e5fe640 10 monclient: init 2024-03-25T19:35:41.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.744+0000 7f210e5fe640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:41.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.744+0000 7f210e5fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:41.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.744+0000 7f210e5fe640 10 monclient(hunting): picked mon.noname-b con 0x7f21080a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:41.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.744+0000 7f210e5fe640 10 monclient(hunting): picked mon.noname-c con 0x7f210813f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:41.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.744+0000 7f210e5fe640 10 monclient(hunting): picked mon.noname-a con 0x7f21081430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:41.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.744+0000 7f210e5fe640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2108142520 con 0x7f21081430c0 2024-03-25T19:35:41.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.744+0000 7f210e5fe640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21080b28a0 con 0x7f210813f710 2024-03-25T19:35:41.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.744+0000 7f210e5fe640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21080b2a30 con 0x7f21080a98f0 2024-03-25T19:35:41.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.744+0000 7f210e5fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:41.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.744+0000 7f210e5fe640 10 monclient(hunting): authenticate will time out at 6395.014293s 2024-03-25T19:35:41.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.748+0000 7f2107fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f21080a98f0 0x7f21080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:47880/0 (socket says 172.21.15.19:47880) 2024-03-25T19:35:41.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.748+0000 7f2107fff640 1 -- 172.21.15.19:0/2682139900 learned_addr learned my addr 172.21.15.19:0/2682139900 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:41.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.756+0000 7f2106ffd640 1 -- 172.21.15.19:0/2682139900 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331625175 0 0) 0x7f21080b2a30 con 0x7f21080a98f0 2024-03-25T19:35:41.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.756+0000 7f2106ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.756+0000 7f2106ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.756+0000 7f2106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.756+0000 7f2106ffd640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:35:41.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.756+0000 7f2106ffd640 1 -- 172.21.15.19:0/2682139900 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f0003520 con 0x7f21080a98f0 2024-03-25T19:35:41.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.763+0000 7f2106ffd640 1 -- 172.21.15.19:0/2682139900 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836596927 0 0) 0x7f2108142520 con 0x7f21081430c0 2024-03-25T19:35:41.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.763+0000 7f2106ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.763+0000 7f2106ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.763+0000 7f2106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.763+0000 7f2106ffd640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:35:41.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.763+0000 7f2106ffd640 1 -- 172.21.15.19:0/2682139900 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21080b2a30 con 0x7f21081430c0 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 1 -- 172.21.15.19:0/2682139900 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2136118733 0 0) 0x7f21080b2a30 con 0x7f21081430c0 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 1 -- 172.21.15.19:0/2682139900 >> v1:172.21.15.19:6790/0 conn(0x7f21080a98f0 legacy=0x7f21080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 1 -- 172.21.15.19:0/2682139900 >> v1:172.21.15.188:6789/0 conn(0x7f210813f710 legacy=0x7f21081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 1 monclient: found mon.noname-a 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 1 -- 172.21.15.19:0/2682139900 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21080b44c0 con 0x7f21081430c0 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 1 -- 172.21.15.19:0/2682139900 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f21000019f0 con 0x7f21081430c0 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f2106ffd640 10 monclient: dump: 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f210e5fe640 5 monclient: authenticate success, global_id 4527 2024-03-25T19:35:41.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.772+0000 7f210e5fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:41.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.787+0000 7f2106ffd640 1 -- 172.21.15.19:0/2682139900 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2100001b50 con 0x7f21081430c0 2024-03-25T19:35:41.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.787+0000 7f2106ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:41.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.787+0000 7f2106ffd640 1 -- 172.21.15.19:0/2682139900 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2100001ee0 con 0x7f21081430c0 2024-03-25T19:35:41.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.787+0000 7f2106ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.787+0000 7f2106ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.787+0000 7f2106ffd640 10 monclient: dump: 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.787+0000 7f210e5fe640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:41.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.787+0000 7f210e5fe640 4 set_mon_vals no callback set 2024-03-25T19:35:41.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.788+0000 7f210e5fe640 10 monclient: shutdown 2024-03-25T19:35:41.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.788+0000 7f210e5fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:41.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.788+0000 7f210e5fe640 1 -- 172.21.15.19:0/2682139900 >> v1:172.21.15.19:6789/0 conn(0x7f21081430c0 legacy=0x7f210814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.788+0000 7f210e5fe640 1 -- 172.21.15.19:0/2682139900 shutdown_connections 2024-03-25T19:35:41.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.788+0000 7f210e5fe640 1 -- 172.21.15.19:0/2682139900 >> 172.21.15.19:0/2682139900 conn(0x7f21080b7ea0 msgr2=0x7f21080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:41.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.788+0000 7f210e5fe640 1 -- 172.21.15.19:0/2682139900 shutdown_connections 2024-03-25T19:35:41.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.788+0000 7f210e5fe640 1 -- 172.21.15.19:0/2682139900 wait complete. 2024-03-25T19:35:41.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.789+0000 7f210e5fe640 10 monclient: build_initial_monmap 2024-03-25T19:35:41.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.789+0000 7f210e5fe640 10 monclient: monmap: 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.789+0000 7f210e5fe640 1 librados: starting msgr at 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.789+0000 7f210e5fe640 1 librados: starting objecter 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.790+0000 7f210e5fe640 1 Processor -- start 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.790+0000 7f210e5fe640 1 -- start start 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.790+0000 7f210e5fe640 1 librados: setting wanted keys 2024-03-25T19:35:41.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.790+0000 7f210e5fe640 1 librados: calling monclient init 2024-03-25T19:35:41.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.790+0000 7f210e5fe640 10 monclient: init 2024-03-25T19:35:41.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.791+0000 7f210e5fe640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:41.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.791+0000 7f210e5fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:41.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.791+0000 7f210e5fe640 10 monclient(hunting): picked mon.noname-b con 0x7f21080a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:41.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.791+0000 7f210e5fe640 10 monclient(hunting): picked mon.noname-a con 0x7f210813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:41.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.791+0000 7f210e5fe640 10 monclient(hunting): picked mon.noname-c con 0x7f21081430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:41.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.791+0000 7f210e5fe640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2108139910 con 0x7f210813f710 2024-03-25T19:35:41.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.791+0000 7f210e5fe640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21080ab8b0 con 0x7f21080a98f0 2024-03-25T19:35:41.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.791+0000 7f210e5fe640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21081db470 con 0x7f21081430c0 2024-03-25T19:35:41.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.791+0000 7f210e5fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:41.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.791+0000 7f210e5fe640 10 monclient(hunting): authenticate will time out at 6395.061177s 2024-03-25T19:35:41.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.802+0000 7f2107fff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f21080a98f0 0x7f21081daa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:43514/0 (socket says 172.21.15.19:43514) 2024-03-25T19:35:41.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.802+0000 7f2107fff640 1 -- 172.21.15.19:0/314780636 learned_addr learned my addr 172.21.15.19:0/314780636 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:41.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.808+0000 7f2104ff9640 10 client.?.objecter ms_handle_connect 0x7f21081430c0 2024-03-25T19:35:41.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.808+0000 7f2104ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.812+0000 7f2104ff9640 10 client.?.objecter ms_handle_connect 0x7f210813f710 2024-03-25T19:35:41.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.812+0000 7f2104ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.817+0000 7f2104ff9640 10 client.?.objecter ms_handle_connect 0x7f21080a98f0 2024-03-25T19:35:41.817 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.817+0000 7f2104ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.818 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.818+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540640830 0 0) 0x7f2108139910 con 0x7f210813f710 2024-03-25T19:35:41.818 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.818+0000 7f2104ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.818 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.818+0000 7f2104ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.818 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.818+0000 7f2104ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.818 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.818+0000 7f2104ff9640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:35:41.818 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.818+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f4004eb0 con 0x7f210813f710 2024-03-25T19:35:41.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.825+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306933421 0 0) 0x7f21081db470 con 0x7f21081430c0 2024-03-25T19:35:41.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.825+0000 7f2104ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.825+0000 7f2104ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.825+0000 7f2104ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.825+0000 7f2104ff9640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:35:41.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.825+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2108139910 con 0x7f21081430c0 2024-03-25T19:35:41.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.826+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623881178 0 0) 0x7f21080ab8b0 con 0x7f21080a98f0 2024-03-25T19:35:41.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.826+0000 7f2104ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.826+0000 7f2104ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.826+0000 7f2104ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.826+0000 7f2104ff9640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:35:41.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.826+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21081db470 con 0x7f21080a98f0 2024-03-25T19:35:41.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.834+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2512099248 0 0) 0x7f20f4004eb0 con 0x7f210813f710 2024-03-25T19:35:41.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.834+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21080ab8b0 con 0x7f210813f710 2024-03-25T19:35:41.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.834+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f20fc001930 con 0x7f210813f710 2024-03-25T19:35:41.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.834+0000 7f2104ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.834+0000 7f2104ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:41.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.834+0000 7f2104ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:41.835 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:41.835 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:41.835 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:41.836 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:41.836 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.836 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:41.836 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:41.836 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:41.836 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:41.836 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:41.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.834+0000 7f2104ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:41.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.841+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2101159080 0 0) 0x7f2108139910 con 0x7f21081430c0 2024-03-25T19:35:41.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.841+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20f4004eb0 con 0x7f21081430c0 2024-03-25T19:35:41.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.841+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2100001e60 con 0x7f21081430c0 2024-03-25T19:35:41.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.841+0000 7f2104ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.841+0000 7f2104ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:41.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.841+0000 7f2104ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:41.841 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:41.841 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:41.841 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:41.842 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:41.842 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.842 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:41.842 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:41.842 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:41.842 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:41.842 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:41.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1981918142 0 0) 0x7f21080ab8b0 con 0x7f210813f710 2024-03-25T19:35:41.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 >> v1:172.21.15.19:6790/0 conn(0x7f21081430c0 legacy=0x7f2108137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 >> v1:172.21.15.188:6789/0 conn(0x7f21080a98f0 legacy=0x7f21081daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f2104ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f2104ff9640 1 monclient: found mon.a 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f2104ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f2104ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21081db110 con 0x7f210813f710 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f2104ff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f2104ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f2104ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f210e5fe640 5 monclient: authenticate success, global_id 4530 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f210e5fe640 10 monclient: _renew_subs 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f210e5fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f210e5fe640 1 -- 172.21.15.19:0/314780636 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21080b0d50 con 0x7f210813f710 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f210e5fe640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f210e5fe640 10 monclient: _renew_subs 2024-03-25T19:35:41.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f210e5fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:41.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f210e5fe640 1 -- 172.21.15.19:0/314780636 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21080b0930 con 0x7f210813f710 2024-03-25T19:35:41.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.844+0000 7f210e5fe640 1 librados: init done 2024-03-25T19:35:41.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.845+0000 7f210e5fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.845+0000 7f210e5fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.845+0000 7f210e5fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:41.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:41.845+0000 7f210e5fe640 1 -- 172.21.15.19:0/314780636 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21080b10f0 con 0x7f210813f710 2024-03-25T19:35:42.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.172+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20fc0035f0 con 0x7f210813f710 2024-03-25T19:35:42.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.172+0000 7f2104ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.172+0000 7f21067fc640 4 set_mon_vals no callback set 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.172+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f20fc003900 con 0x7f210813f710 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.172+0000 7f2104ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.172+0000 7f2104ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.172+0000 7f2104ff9640 10 monclient: dump: 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:42.173 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:42.174 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:42.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.173+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f20fc0198d0 con 0x7f210813f710 2024-03-25T19:35:42.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.173+0000 7f2107fff640 10 monclient: get_auth_request con 0x7f20f406df70 auth_method 0 2024-03-25T19:35:42.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.174+0000 7f2104ff9640 10 client.4530.objecter ms_handle_connect 0x7f20f406df70 2024-03-25T19:35:42.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.181+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f20fc053690 con 0x7f210813f710 2024-03-25T19:35:42.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.181+0000 7f2104ff9640 10 client.4530.objecter ms_dispatch 0x7f2108000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:42.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.181+0000 7f2104ff9640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:42.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.181+0000 7f2104ff9640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:42.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.181+0000 7f2104ff9640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-25T19:35:42.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.181+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20fc04efd0 con 0x7f210813f710 2024-03-25T19:35:42.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.181+0000 7f2104ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:42.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.181+0000 7f2104ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:42.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.415+0000 7f210e5fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:42.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.415+0000 7f210e5fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:42.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.415+0000 7f210e5fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:42.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.415+0000 7f210e5fe640 1 -- 172.21.15.19:0/314780636 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21081db600 con 0x7f210813f710 2024-03-25T19:35:42.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.448+0000 7f2104ff9640 1 -- 172.21.15.19:0/314780636 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f20fc0510d0 con 0x7f210813f710 2024-03-25T19:35:42.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.448+0000 7f2104ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:42.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.448+0000 7f2104ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:42.448 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:35:42.449 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:42.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.450+0000 7f210e5fe640 10 librados: watch_flush enter 2024-03-25T19:35:42.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.450+0000 7f210e5fe640 10 librados: watch_flush exit 2024-03-25T19:35:42.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.450+0000 7f210e5fe640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-25T19:35:42.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.450+0000 7f210e5fe640 10 client.4530.objecter successfully canceled tick 2024-03-25T19:35:42.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.450+0000 7f210e5fe640 1 -- 172.21.15.19:0/314780636 >> v1:172.21.15.188:6800/2207506229 conn(0x7f20f406df70 legacy=0x7f20f4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:42.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.450+0000 7f210e5fe640 10 monclient: shutdown 2024-03-25T19:35:42.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.450+0000 7f210e5fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:42.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.450+0000 7f210e5fe640 1 -- 172.21.15.19:0/314780636 >> v1:172.21.15.19:6789/0 conn(0x7f210813f710 legacy=0x7f21080b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:42.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.450+0000 7f210e5fe640 1 -- 172.21.15.19:0/314780636 shutdown_connections 2024-03-25T19:35:42.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.450+0000 7f210e5fe640 1 -- 172.21.15.19:0/314780636 >> 172.21.15.19:0/314780636 conn(0x7f21080b7ea0 msgr2=0x7f2108074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:42.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.450+0000 7f210e5fe640 1 -- 172.21.15.19:0/314780636 shutdown_connections 2024-03-25T19:35:42.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.451+0000 7f210e5fe640 1 -- 172.21.15.19:0/314780636 wait complete. 2024-03-25T19:35:42.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:42.451+0000 7f210e5fe640 1 librados: shutdown 2024-03-25T19:35:42.463 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:49.270 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:49.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.385+0000 7f7425e72640 10 monclient: get_monmap_and_config 2024-03-25T19:35:49.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.385+0000 7f7425e72640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.385+0000 7f7425e72640 10 monclient: monmap: 2024-03-25T19:35:49.386 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:49.386 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:49.386 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:35:49.386869+0000 2024-03-25T19:35:49.386 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:35:49.386869+0000 2024-03-25T19:35:49.386 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.386 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:49.386 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:49.386 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:35:49.386 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:35:49.386 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.386+0000 7f7425e72640 1 Processor -- start 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.386+0000 7f7425e72640 1 -- start start 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.386+0000 7f7425e72640 10 monclient: init 2024-03-25T19:35:49.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.387+0000 7f7425e72640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.387+0000 7f7425e72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.387+0000 7f7425e72640 10 monclient(hunting): picked mon.noname-c con 0x7f74200ae6e0 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.387+0000 7f7425e72640 10 monclient(hunting): picked mon.noname-b con 0x7f742006f080 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.387+0000 7f7425e72640 10 monclient(hunting): picked mon.noname-a con 0x7f74200704e0 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.387+0000 7f7425e72640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74200718d0 con 0x7f74200704e0 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.387+0000 7f7425e72640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742006fa30 con 0x7f74200ae6e0 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.387+0000 7f7425e72640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7420070090 con 0x7f742006f080 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.387+0000 7f7425e72640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.387+0000 7f7425e72640 10 monclient(hunting): authenticate will time out at 6402.657383s 2024-03-25T19:35:49.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.391+0000 7f741f7fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f74200ae6e0 0x7f742013c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:48362/0 (socket says 172.21.15.19:48362) 2024-03-25T19:35:49.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.391+0000 7f741f7fe640 1 -- 172.21.15.19:0/4086956364 learned_addr learned my addr 172.21.15.19:0/4086956364 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:49.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.409+0000 7f741e7fc640 1 -- 172.21.15.19:0/4086956364 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507310969 0 0) 0x7f742006fa30 con 0x7f74200ae6e0 2024-03-25T19:35:49.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.409+0000 7f741e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.409+0000 7f741e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.409+0000 7f741e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.409+0000 7f741e7fc640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:35:49.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.409+0000 7f741e7fc640 1 -- 172.21.15.19:0/4086956364 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7408003520 con 0x7f74200ae6e0 2024-03-25T19:35:49.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.413+0000 7f741e7fc640 1 -- 172.21.15.19:0/4086956364 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161453699 0 0) 0x7f7420070090 con 0x7f742006f080 2024-03-25T19:35:49.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.413+0000 7f741e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.413+0000 7f741e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.413+0000 7f741e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.413+0000 7f741e7fc640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:35:49.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.413+0000 7f741e7fc640 1 -- 172.21.15.19:0/4086956364 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f742006fa30 con 0x7f742006f080 2024-03-25T19:35:49.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 1 -- 172.21.15.19:0/4086956364 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1294306726 0 0) 0x7f7408003520 con 0x7f74200ae6e0 2024-03-25T19:35:49.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 1 -- 172.21.15.19:0/4086956364 >> v1:172.21.15.19:6790/0 conn(0x7f742006f080 legacy=0x7f7420071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 1 -- 172.21.15.19:0/4086956364 >> v1:172.21.15.19:6789/0 conn(0x7f74200704e0 legacy=0x7f742014a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 1 -- 172.21.15.19:0/4086956364 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742006f510 con 0x7f74200ae6e0 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 1 -- 172.21.15.19:0/4086956364 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7410001690 con 0x7f74200ae6e0 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f7425e72640 5 monclient: authenticate success, global_id 4519 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f7425e72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:49.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f741e7fc640 10 monclient: dump: 2024-03-25T19:35:49.422 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:49.422 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:49.422 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:49.422 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:49.422 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.422 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:49.422 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:49.422 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:49.422 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:49.422 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:49.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.420+0000 7f7425e72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:49.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.425+0000 7f741e7fc640 1 -- 172.21.15.19:0/4086956364 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74100017f0 con 0x7f74200ae6e0 2024-03-25T19:35:49.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.425+0000 7f741e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.425+0000 7f7425e72640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:49.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.425+0000 7f7425e72640 4 set_mon_vals no callback set 2024-03-25T19:35:49.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.426+0000 7f7425e72640 10 monclient: shutdown 2024-03-25T19:35:49.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.426+0000 7f7425e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.426+0000 7f7425e72640 1 -- 172.21.15.19:0/4086956364 >> v1:172.21.15.188:6789/0 conn(0x7f74200ae6e0 legacy=0x7f742013c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.426+0000 7f7425e72640 1 -- 172.21.15.19:0/4086956364 shutdown_connections 2024-03-25T19:35:49.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.426+0000 7f7425e72640 1 -- 172.21.15.19:0/4086956364 >> 172.21.15.19:0/4086956364 conn(0x7f742006b630 msgr2=0x7f74200b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.426+0000 7f7425e72640 1 -- 172.21.15.19:0/4086956364 shutdown_connections 2024-03-25T19:35:49.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.426+0000 7f7425e72640 1 -- 172.21.15.19:0/4086956364 wait complete. 2024-03-25T19:35:49.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.426+0000 7f7425e72640 10 monclient: build_initial_monmap 2024-03-25T19:35:49.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.426+0000 7f7425e72640 10 monclient: monmap: 2024-03-25T19:35:49.428 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:49.428 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:49.428 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:35:49.428 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:35:49.428 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:49.428 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:35:49.428 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:49.428 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:35:49.428 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:35:49.428 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.427+0000 7f7425e72640 1 librados: starting msgr at 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.427+0000 7f7425e72640 1 librados: starting objecter 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.427+0000 7f7425e72640 1 Processor -- start 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.427+0000 7f7425e72640 1 -- start start 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.427+0000 7f7425e72640 1 librados: setting wanted keys 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.427+0000 7f7425e72640 1 librados: calling monclient init 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.427+0000 7f7425e72640 10 monclient: init 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.428+0000 7f7425e72640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.428+0000 7f7425e72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.428+0000 7f7425e72640 10 monclient(hunting): picked mon.noname-a con 0x7f74200704e0 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.428+0000 7f7425e72640 10 monclient(hunting): picked mon.noname-c con 0x7f74200ae6e0 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.428+0000 7f7425e72640 10 monclient(hunting): picked mon.noname-b con 0x7f7420140240 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.428+0000 7f7425e72640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74200a9800 con 0x7f74200704e0 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.428+0000 7f7425e72640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7420142170 con 0x7f7420140240 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.428+0000 7f7425e72640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74200a9a30 con 0x7f74200ae6e0 2024-03-25T19:35:49.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.428+0000 7f7425e72640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:49.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.428+0000 7f7425e72640 10 monclient(hunting): authenticate will time out at 6402.698043s 2024-03-25T19:35:49.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.439+0000 7f741ffff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f7420140240 0x7f74201e1e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:48396/0 (socket says 172.21.15.19:48396) 2024-03-25T19:35:49.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.439+0000 7f741ffff640 1 -- 172.21.15.19:0/418416171 learned_addr learned my addr 172.21.15.19:0/418416171 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:49.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.440+0000 7f7407fff640 10 client.?.objecter ms_handle_connect 0x7f7420140240 2024-03-25T19:35:49.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.440+0000 7f7407fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.441+0000 7f7407fff640 10 client.?.objecter ms_handle_connect 0x7f74200ae6e0 2024-03-25T19:35:49.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.441+0000 7f7407fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.446+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077664163 0 0) 0x7f74200a9a30 con 0x7f74200ae6e0 2024-03-25T19:35:49.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.446+0000 7f7407fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.446+0000 7f7407fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.446+0000 7f7407fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.446+0000 7f7407fff640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:35:49.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.446+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f740c004eb0 con 0x7f74200ae6e0 2024-03-25T19:35:49.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.448+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690341482 0 0) 0x7f7420142170 con 0x7f7420140240 2024-03-25T19:35:49.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.448+0000 7f7407fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.448+0000 7f7407fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.448+0000 7f7407fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.448+0000 7f7407fff640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:35:49.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.448+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74200a9a30 con 0x7f7420140240 2024-03-25T19:35:49.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.454+0000 7f7407fff640 10 client.?.objecter ms_handle_connect 0x7f74200704e0 2024-03-25T19:35:49.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.454+0000 7f7407fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:49.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.456+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2460736075 0 0) 0x7f740c004eb0 con 0x7f74200ae6e0 2024-03-25T19:35:49.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.456+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7420142170 con 0x7f74200ae6e0 2024-03-25T19:35:49.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.457+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f74140019f0 con 0x7f74200ae6e0 2024-03-25T19:35:49.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.457+0000 7f7407fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.457+0000 7f7407fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:49.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.457+0000 7f7407fff640 10 monclient(hunting): dump: 2024-03-25T19:35:49.458 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:49.458 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:49.458 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:49.458 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:49.458 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.458 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:49.458 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:49.458 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:49.458 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:49.458 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:49.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.457+0000 7f7407fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:49.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.459+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1490690785 0 0) 0x7f74200a9a30 con 0x7f7420140240 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.459+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f740c004eb0 con 0x7f7420140240 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.459+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7418001c90 con 0x7f7420140240 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.459+0000 7f7407fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.459+0000 7f7407fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.459+0000 7f7407fff640 10 monclient(hunting): dump: 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:49.460 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:49.461 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:49.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.461+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705836219 0 0) 0x7f74200a9800 con 0x7f74200704e0 2024-03-25T19:35:49.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.461+0000 7f7407fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:49.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.461+0000 7f7407fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:49.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.461+0000 7f7407fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:49.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.461+0000 7f7407fff640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:35:49.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.461+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74200a9a30 con 0x7f74200704e0 2024-03-25T19:35:49.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2646491388 0 0) 0x7f7420142170 con 0x7f74200ae6e0 2024-03-25T19:35:49.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 >> v1:172.21.15.188:6789/0 conn(0x7f7420140240 legacy=0x7f74201e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 >> v1:172.21.15.19:6789/0 conn(0x7f74200704e0 legacy=0x7f7420141860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7407fff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:49.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7407fff640 1 monclient: found mon.c 2024-03-25T19:35:49.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7407fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:49.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7407fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:35:49.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7420142d60 con 0x7f74200ae6e0 2024-03-25T19:35:49.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7407fff640 10 monclient: _finish_auth 0 2024-03-25T19:35:49.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7407fff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:49.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7407fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:49.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7425e72640 5 monclient: authenticate success, global_id 4505 2024-03-25T19:35:49.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7425e72640 10 monclient: _renew_subs 2024-03-25T19:35:49.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7425e72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:35:49.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7425e72640 1 -- 172.21.15.19:0/418416171 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74200aef70 con 0x7f74200ae6e0 2024-03-25T19:35:49.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7425e72640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:49.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7425e72640 10 monclient: _renew_subs 2024-03-25T19:35:49.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7425e72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:35:49.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7425e72640 1 -- 172.21.15.19:0/418416171 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74201e3570 con 0x7f74200ae6e0 2024-03-25T19:35:49.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.463+0000 7f7425e72640 1 librados: init done 2024-03-25T19:35:49.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.464+0000 7f7425e72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.464+0000 7f7425e72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.464+0000 7f7425e72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:35:49.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.464+0000 7f7425e72640 1 -- 172.21.15.19:0/418416171 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74200627b0 con 0x7f74200ae6e0 2024-03-25T19:35:49.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.776+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7414003610 con 0x7f74200ae6e0 2024-03-25T19:35:49.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.776+0000 7f7407fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.776+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7414003920 con 0x7f74200ae6e0 2024-03-25T19:35:49.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.776+0000 7f7407fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.776+0000 7f7407fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:49.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.776+0000 7f7407fff640 10 monclient: dump: 2024-03-25T19:35:49.777 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:49.777 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:49.777 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:49.777 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:49.777 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.777 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:49.777 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:49.778 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:49.778 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:49.778 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:49.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.776+0000 7f741dffb640 4 set_mon_vals no callback set 2024-03-25T19:35:49.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.777+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f7414019930 con 0x7f74200ae6e0 2024-03-25T19:35:49.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.778+0000 7f741f7fe640 10 monclient: get_auth_request con 0x7f740c06df00 auth_method 0 2024-03-25T19:35:49.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.778+0000 7f7407fff640 10 client.4505.objecter ms_handle_connect 0x7f740c06df00 2024-03-25T19:35:49.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.784+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f74140535f0 con 0x7f74200ae6e0 2024-03-25T19:35:49.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.784+0000 7f7407fff640 10 client.4505.objecter ms_dispatch 0x7f7420000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:49.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.784+0000 7f7407fff640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:49.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.784+0000 7f7407fff640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:49.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.784+0000 7f7407fff640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-25T19:35:49.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.784+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f741404ef30 con 0x7f74200ae6e0 2024-03-25T19:35:49.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.784+0000 7f7407fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:49.784+0000 7f7407fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:50.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.042+0000 7f7425e72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:50.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.042+0000 7f7425e72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:50.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.042+0000 7f7425e72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:35:50.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.042+0000 7f7425e72640 1 -- 172.21.15.19:0/418416171 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74200681e0 con 0x7f74200ae6e0 2024-03-25T19:35:50.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.173+0000 7f7407fff640 1 -- 172.21.15.19:0/418416171 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f7414051030 con 0x7f74200ae6e0 2024-03-25T19:35:50.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.173+0000 7f7407fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:50.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.173+0000 7f7407fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:50.174 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:35:50.175 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:50.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.176+0000 7f7425e72640 10 librados: watch_flush enter 2024-03-25T19:35:50.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.176+0000 7f7425e72640 10 librados: watch_flush exit 2024-03-25T19:35:50.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.176+0000 7f7425e72640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-25T19:35:50.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.176+0000 7f7425e72640 10 client.4505.objecter successfully canceled tick 2024-03-25T19:35:50.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.176+0000 7f7425e72640 1 -- 172.21.15.19:0/418416171 >> v1:172.21.15.188:6800/2207506229 conn(0x7f740c06df00 legacy=0x7f740c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.176+0000 7f7425e72640 10 monclient: shutdown 2024-03-25T19:35:50.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.176+0000 7f7425e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:50.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.176+0000 7f7425e72640 1 -- 172.21.15.19:0/418416171 >> v1:172.21.15.19:6790/0 conn(0x7f74200ae6e0 legacy=0x7f7420142650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:50.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.176+0000 7f7425e72640 1 -- 172.21.15.19:0/418416171 shutdown_connections 2024-03-25T19:35:50.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.176+0000 7f7425e72640 1 -- 172.21.15.19:0/418416171 >> 172.21.15.19:0/418416171 conn(0x7f742006b630 msgr2=0x7f742013c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:50.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.177+0000 7f7425e72640 1 -- 172.21.15.19:0/418416171 shutdown_connections 2024-03-25T19:35:50.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.177+0000 7f7425e72640 1 -- 172.21.15.19:0/418416171 wait complete. 2024-03-25T19:35:50.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:50.177+0000 7f7425e72640 1 librados: shutdown 2024-03-25T19:35:50.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:52.891 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:07:15.620 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:52.938 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:07:15.641 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:52.942 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:07:15.672 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:52.943 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:07:15.645 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:52.971 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:07:15.674 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:52.988 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:07:15.717 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:53.008 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:07:15.738 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:53.013 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:07:15.743 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:53.045 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:07:15.775 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:56.993 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:57.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.091+0000 7f393bcb3640 10 monclient: get_monmap_and_config 2024-03-25T19:35:57.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.091+0000 7f393bcb3640 10 monclient: build_initial_monmap 2024-03-25T19:35:57.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.091+0000 7f393bcb3640 10 monclient: monmap: 2024-03-25T19:35:57.092 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:57.092 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:57.092 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:35:57.092771+0000 2024-03-25T19:35:57.092 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:35:57.092771+0000 2024-03-25T19:35:57.092 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:57.093 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:57.093 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:57.093 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:35:57.093 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:35:57.093 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:57.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 1 Processor -- start 2024-03-25T19:35:57.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 1 -- start start 2024-03-25T19:35:57.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 10 monclient: init 2024-03-25T19:35:57.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:57.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:57.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 10 monclient(hunting): picked mon.noname-b con 0x7f39340b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:57.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 10 monclient(hunting): picked mon.noname-c con 0x7f39340b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:57.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 10 monclient(hunting): picked mon.noname-a con 0x7f393414be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:57.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39340af940 con 0x7f393414be90 2024-03-25T19:35:57.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39340b0a70 con 0x7f39340b1730 2024-03-25T19:35:57.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39340b0d50 con 0x7f39340b4d60 2024-03-25T19:35:57.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:57.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.092+0000 7f393bcb3640 10 monclient(hunting): authenticate will time out at 6410.362861s 2024-03-25T19:35:57.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.093+0000 7f3939a4f640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f39340b4d60 0x7f393413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:56370/0 (socket says 172.21.15.19:56370) 2024-03-25T19:35:57.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.093+0000 7f393924e640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f39340b1730 0x7f39340b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:45278/0 (socket says 172.21.15.19:45278) 2024-03-25T19:35:57.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.093+0000 7f3939a4f640 1 -- 172.21.15.19:0/3528429503 learned_addr learned my addr 172.21.15.19:0/3528429503 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:57.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.119+0000 7f3938a4d640 1 -- 172.21.15.19:0/3528429503 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83570086 0 0) 0x7f39340af940 con 0x7f393414be90 2024-03-25T19:35:57.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.119+0000 7f3938a4d640 10 monclient(hunting): _negotiate 2024-03-25T19:35:57.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.119+0000 7f3938a4d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:57.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.119+0000 7f3938a4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:57.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.119+0000 7f3938a4d640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:35:57.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.119+0000 7f3938a4d640 1 -- 172.21.15.19:0/3528429503 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3920003520 con 0x7f393414be90 2024-03-25T19:35:57.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.127+0000 7f3938a4d640 1 -- 172.21.15.19:0/3528429503 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2245612827 0 0) 0x7f3920003520 con 0x7f393414be90 2024-03-25T19:35:57.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 1 -- 172.21.15.19:0/3528429503 >> v1:172.21.15.19:6790/0 conn(0x7f39340b4d60 legacy=0x7f393413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 1 -- 172.21.15.19:0/3528429503 >> v1:172.21.15.188:6789/0 conn(0x7f39340b1730 legacy=0x7f39340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 10 monclient: _finish_hunting 0 2024-03-25T19:35:57.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 1 monclient: found mon.noname-a 2024-03-25T19:35:57.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:57.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 1 -- 172.21.15.19:0/3528429503 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39340b1030 con 0x7f393414be90 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 10 monclient: _finish_auth 0 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 10 monclient: _check_auth_tickets 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 1 -- 172.21.15.19:0/3528429503 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f392c001a10 con 0x7f393414be90 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f3938a4d640 10 monclient: dump: 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:57.129 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:57.130 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:57.130 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:57.130 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:57.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f393bcb3640 5 monclient: authenticate success, global_id 4539 2024-03-25T19:35:57.130 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.128+0000 7f393bcb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:57.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.138+0000 7f3938a4d640 1 -- 172.21.15.19:0/3528429503 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f392c001b70 con 0x7f393414be90 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.138+0000 7f3938a4d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.138+0000 7f3938a4d640 1 -- 172.21.15.19:0/3528429503 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f392c001f80 con 0x7f393414be90 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.138+0000 7f393bcb3640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.138+0000 7f393bcb3640 4 set_mon_vals no callback set 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.138+0000 7f3938a4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.138+0000 7f3938a4d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.138+0000 7f3938a4d640 10 monclient: dump: 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:57.139 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:57.140 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:57.140 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:57.140 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:57.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.139+0000 7f393bcb3640 10 monclient: shutdown 2024-03-25T19:35:57.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.139+0000 7f393bcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:57.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.139+0000 7f393bcb3640 1 -- 172.21.15.19:0/3528429503 >> v1:172.21.15.19:6789/0 conn(0x7f393414be90 legacy=0x7f393414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.139+0000 7f393bcb3640 1 -- 172.21.15.19:0/3528429503 shutdown_connections 2024-03-25T19:35:57.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.139+0000 7f393bcb3640 1 -- 172.21.15.19:0/3528429503 >> 172.21.15.19:0/3528429503 conn(0x7f39340617a0 msgr2=0x7f3934061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:57.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.139+0000 7f393bcb3640 1 -- 172.21.15.19:0/3528429503 shutdown_connections 2024-03-25T19:35:57.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.139+0000 7f393bcb3640 1 -- 172.21.15.19:0/3528429503 wait complete. 2024-03-25T19:35:57.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.140+0000 7f393bcb3640 10 monclient: build_initial_monmap 2024-03-25T19:35:57.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.140+0000 7f393bcb3640 10 monclient: monmap: 2024-03-25T19:35:57.142 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:35:57.142 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 1 librados: starting msgr at 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 1 librados: starting objecter 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 1 Processor -- start 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 1 -- start start 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 1 librados: setting wanted keys 2024-03-25T19:35:57.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 1 librados: calling monclient init 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 10 monclient: init 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 10 monclient(hunting): picked mon.noname-c con 0x7f39340b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 10 monclient(hunting): picked mon.noname-b con 0x7f39340b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 10 monclient(hunting): picked mon.noname-a con 0x7f393414be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3934137760 con 0x7f393414be90 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39341387d0 con 0x7f39340b4d60 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3934140380 con 0x7f39340b1730 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.141+0000 7f393bcb3640 10 monclient(hunting): authenticate will time out at 6410.411740s 2024-03-25T19:35:57.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.157+0000 7f393a250640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f393414be90 0x7f39341e2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:48198/0 (socket says 172.21.15.19:48198) 2024-03-25T19:35:57.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.157+0000 7f393a250640 1 -- 172.21.15.19:0/359786636 learned_addr learned my addr 172.21.15.19:0/359786636 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:35:57.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.159+0000 7f392a7fc640 10 client.?.objecter ms_handle_connect 0x7f393414be90 2024-03-25T19:35:57.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.159+0000 7f392a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:57.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.169+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728123441 0 0) 0x7f3934137760 con 0x7f393414be90 2024-03-25T19:35:57.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.170+0000 7f392a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:57.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.170+0000 7f392a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:57.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.170+0000 7f392a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:57.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.170+0000 7f392a7fc640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:35:57.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.170+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3918004eb0 con 0x7f393414be90 2024-03-25T19:35:57.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.172+0000 7f392a7fc640 10 client.?.objecter ms_handle_connect 0x7f39340b4d60 2024-03-25T19:35:57.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.172+0000 7f392a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:57.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.186+0000 7f392a7fc640 10 client.?.objecter ms_handle_connect 0x7f39340b1730 2024-03-25T19:35:57.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.186+0000 7f392a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:57.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.190+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 454303273 0 0) 0x7f3918004eb0 con 0x7f393414be90 2024-03-25T19:35:57.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.190+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3934137760 con 0x7f393414be90 2024-03-25T19:35:57.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.190+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f392c001fc0 con 0x7f393414be90 2024-03-25T19:35:57.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.190+0000 7f392a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:57.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.190+0000 7f392a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:57.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.190+0000 7f392a7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:57.190 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:57.191 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:57.191 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:57.191 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:57.191 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:57.191 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:57.191 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:57.191 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:57.191 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:57.191 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:57.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.190+0000 7f392a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3892292715 0 0) 0x7f3934137760 con 0x7f393414be90 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 >> v1:172.21.15.19:6790/0 conn(0x7f39340b1730 legacy=0x7f3934140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 >> v1:172.21.15.188:6789/0 conn(0x7f39340b4d60 legacy=0x7f3934144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f392a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f392a7fc640 1 monclient: found mon.a 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f392a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f392a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3934136910 con 0x7f393414be90 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f392a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f392a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f392a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f393bcb3640 5 monclient: authenticate success, global_id 4542 2024-03-25T19:35:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f393bcb3640 10 monclient: _renew_subs 2024-03-25T19:35:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f393bcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f393bcb3640 1 -- 172.21.15.19:0/359786636 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3934136b60 con 0x7f393414be90 2024-03-25T19:35:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f393bcb3640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f393bcb3640 10 monclient: _renew_subs 2024-03-25T19:35:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f393bcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f393bcb3640 1 -- 172.21.15.19:0/359786636 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39341372b0 con 0x7f393414be90 2024-03-25T19:35:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.252+0000 7f393bcb3640 1 librados: init done 2024-03-25T19:35:57.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.253+0000 7f393bcb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:57.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.253+0000 7f393bcb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:57.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.253+0000 7f393bcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:57.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.253+0000 7f393bcb3640 1 -- 172.21.15.19:0/359786636 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3934140510 con 0x7f393414be90 2024-03-25T19:35:57.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.568+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f392c003c20 con 0x7f393414be90 2024-03-25T19:35:57.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.568+0000 7f392a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:57.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.568+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f392c003fb0 con 0x7f393414be90 2024-03-25T19:35:57.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.568+0000 7f392a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:57.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.568+0000 7f392a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:57.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.568+0000 7f392a7fc640 10 monclient: dump: 2024-03-25T19:35:57.568 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:35:57.569 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:35:57.569 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:57.569 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:35:57.569 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:57.569 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:35:57.569 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:35:57.569 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:35:57.569 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:35:57.569 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:35:57.569 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.568+0000 7f392bfff640 4 set_mon_vals no callback set 2024-03-25T19:35:57.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.568+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f392c0041d0 con 0x7f393414be90 2024-03-25T19:35:57.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.569+0000 7f3939a4f640 10 monclient: get_auth_request con 0x7f391806dd00 auth_method 0 2024-03-25T19:35:57.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.569+0000 7f392a7fc640 10 client.4542.objecter ms_handle_connect 0x7f391806dd00 2024-03-25T19:35:57.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.575+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f392c054b10 con 0x7f393414be90 2024-03-25T19:35:57.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.575+0000 7f392a7fc640 10 client.4542.objecter ms_dispatch 0x7f3934000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:57.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.575+0000 7f392a7fc640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:57.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.575+0000 7f392a7fc640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:57.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.575+0000 7f392a7fc640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-25T19:35:57.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.575+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f392c050620 con 0x7f393414be90 2024-03-25T19:35:57.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.575+0000 7f392a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:57.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.575+0000 7f392a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:57.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.833+0000 7f393bcb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:57.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.833+0000 7f393bcb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:57.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.833+0000 7f393bcb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:35:57.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.833+0000 7f393bcb3640 1 -- 172.21.15.19:0/359786636 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39340627b0 con 0x7f393414be90 2024-03-25T19:35:57.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.873+0000 7f392a7fc640 1 -- 172.21.15.19:0/359786636 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f392c052550 con 0x7f393414be90 2024-03-25T19:35:57.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.873+0000 7f392a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:57.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.873+0000 7f392a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:57.873 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:35:57.873 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:57.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 10 librados: watch_flush enter 2024-03-25T19:35:57.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 10 librados: watch_flush exit 2024-03-25T19:35:57.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-25T19:35:57.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 10 client.4542.objecter successfully canceled tick 2024-03-25T19:35:57.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 1 -- 172.21.15.19:0/359786636 >> v1:172.21.15.188:6800/2207506229 conn(0x7f391806dd00 legacy=0x7f39180701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 10 monclient: shutdown 2024-03-25T19:35:57.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:57.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 1 -- 172.21.15.19:0/359786636 >> v1:172.21.15.19:6789/0 conn(0x7f393414be90 legacy=0x7f39341e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 1 -- 172.21.15.19:0/359786636 shutdown_connections 2024-03-25T19:35:57.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 1 -- 172.21.15.19:0/359786636 >> 172.21.15.19:0/359786636 conn(0x7f39340617a0 msgr2=0x7f39340b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:57.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 1 -- 172.21.15.19:0/359786636 shutdown_connections 2024-03-25T19:35:57.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 1 -- 172.21.15.19:0/359786636 wait complete. 2024-03-25T19:35:57.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:35:57.875+0000 7f393bcb3640 1 librados: shutdown 2024-03-25T19:35:57.887 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:04.694 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:04.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.813+0000 7fe8de243640 10 monclient: get_monmap_and_config 2024-03-25T19:36:04.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.814+0000 7fe8de243640 10 monclient: build_initial_monmap 2024-03-25T19:36:04.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.814+0000 7fe8de243640 10 monclient: monmap: 2024-03-25T19:36:04.814 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:04.814 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:04.814 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:36:04.815260+0000 2024-03-25T19:36:04.814 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:36:04.815260+0000 2024-03-25T19:36:04.814 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:04.814 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:04.814 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:04.814 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:36:04.815 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:36:04.815 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:04.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.814+0000 7fe8de243640 1 Processor -- start 2024-03-25T19:36:04.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.814+0000 7fe8de243640 1 -- start start 2024-03-25T19:36:04.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.814+0000 7fe8de243640 10 monclient: init 2024-03-25T19:36:04.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.815+0000 7fe8de243640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:04.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.815+0000 7fe8de243640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:04.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.815+0000 7fe8de243640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d80705f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:04.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.815+0000 7fe8de243640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d80709d0 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:04.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.816+0000 7fe8de243640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d8140a40 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:04.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.816+0000 7fe8de243640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d8071980 con 0x7fe8d80705f0 2024-03-25T19:36:04.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.816+0000 7fe8de243640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d80728b0 con 0x7fe8d8140a40 2024-03-25T19:36:04.817 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.816+0000 7fe8de243640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d813da10 con 0x7fe8d80709d0 2024-03-25T19:36:04.817 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.816+0000 7fe8de243640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:04.817 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.816+0000 7fe8de243640 10 monclient(hunting): authenticate will time out at 6418.085990s 2024-03-25T19:36:04.817 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.817+0000 7fe8d7fff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fe8d8140a40 0x7fe8d814ba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:40474/0 (socket says 172.21.15.19:40474) 2024-03-25T19:36:04.817 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.817+0000 7fe8d7fff640 1 -- 172.21.15.19:0/1014675664 learned_addr learned my addr 172.21.15.19:0/1014675664 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:04.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.823+0000 7fe8d67fc640 1 -- 172.21.15.19:0/1014675664 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810895103 0 0) 0x7fe8d80728b0 con 0x7fe8d8140a40 2024-03-25T19:36:04.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.823+0000 7fe8d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.823+0000 7fe8d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.823+0000 7fe8d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.823+0000 7fe8d67fc640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:36:04.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.823+0000 7fe8d67fc640 1 -- 172.21.15.19:0/1014675664 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c0003520 con 0x7fe8d8140a40 2024-03-25T19:36:04.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.834+0000 7fe8d67fc640 1 -- 172.21.15.19:0/1014675664 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407608970 0 0) 0x7fe8d813da10 con 0x7fe8d80709d0 2024-03-25T19:36:04.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.835+0000 7fe8d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.835+0000 7fe8d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.835+0000 7fe8d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.835+0000 7fe8d67fc640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:36:04.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.835+0000 7fe8d67fc640 1 -- 172.21.15.19:0/1014675664 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d80728b0 con 0x7fe8d80709d0 2024-03-25T19:36:04.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 1 -- 172.21.15.19:0/1014675664 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1793188006 0 0) 0x7fe8c0003520 con 0x7fe8d8140a40 2024-03-25T19:36:04.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 1 -- 172.21.15.19:0/1014675664 >> v1:172.21.15.19:6790/0 conn(0x7fe8d80709d0 legacy=0x7fe8d8140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 1 -- 172.21.15.19:0/1014675664 >> v1:172.21.15.19:6789/0 conn(0x7fe8d80705f0 legacy=0x7fe8d813d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:04.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 1 monclient: found mon.noname-c 2024-03-25T19:36:04.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:04.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:36:04.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 1 -- 172.21.15.19:0/1014675664 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d80b0230 con 0x7fe8d8140a40 2024-03-25T19:36:04.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:04.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:04.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:04.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 1 -- 172.21.15.19:0/1014675664 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe8d0001a10 con 0x7fe8d8140a40 2024-03-25T19:36:04.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8d67fc640 10 monclient: dump: 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8de243640 5 monclient: authenticate success, global_id 4531 2024-03-25T19:36:04.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.838+0000 7fe8de243640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:04.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.847+0000 7fe8d67fc640 1 -- 172.21.15.19:0/1014675664 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8d0001b70 con 0x7fe8d8140a40 2024-03-25T19:36:04.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.847+0000 7fe8d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:04.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.847+0000 7fe8d67fc640 1 -- 172.21.15.19:0/1014675664 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe8d0001f80 con 0x7fe8d8140a40 2024-03-25T19:36:04.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.847+0000 7fe8d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.847+0000 7fe8d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:04.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.847+0000 7fe8d67fc640 10 monclient: dump: 2024-03-25T19:36:04.848 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:04.848 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:04.848 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:04.848 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:04.848 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.848 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:04.848 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:04.849 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:04.849 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:04.849 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:04.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.847+0000 7fe8de243640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:04.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.847+0000 7fe8de243640 4 set_mon_vals no callback set 2024-03-25T19:36:04.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.848+0000 7fe8de243640 10 monclient: shutdown 2024-03-25T19:36:04.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.848+0000 7fe8de243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:04.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.848+0000 7fe8de243640 1 -- 172.21.15.19:0/1014675664 >> v1:172.21.15.188:6789/0 conn(0x7fe8d8140a40 legacy=0x7fe8d814ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.848+0000 7fe8de243640 1 -- 172.21.15.19:0/1014675664 shutdown_connections 2024-03-25T19:36:04.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.848+0000 7fe8de243640 1 -- 172.21.15.19:0/1014675664 >> 172.21.15.19:0/1014675664 conn(0x7fe8d80b5c00 msgr2=0x7fe8d80b6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.848+0000 7fe8de243640 1 -- 172.21.15.19:0/1014675664 shutdown_connections 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.848+0000 7fe8de243640 1 -- 172.21.15.19:0/1014675664 wait complete. 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.849+0000 7fe8de243640 10 monclient: build_initial_monmap 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.849+0000 7fe8de243640 10 monclient: monmap: 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:36:04.851 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:04.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.850+0000 7fe8de243640 1 librados: starting msgr at 2024-03-25T19:36:04.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.850+0000 7fe8de243640 1 librados: starting objecter 2024-03-25T19:36:04.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.850+0000 7fe8de243640 1 Processor -- start 2024-03-25T19:36:04.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.850+0000 7fe8de243640 1 -- start start 2024-03-25T19:36:04.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.850+0000 7fe8de243640 1 librados: setting wanted keys 2024-03-25T19:36:04.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.850+0000 7fe8de243640 1 librados: calling monclient init 2024-03-25T19:36:04.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.850+0000 7fe8de243640 10 monclient: init 2024-03-25T19:36:04.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.851+0000 7fe8de243640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:04.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.851+0000 7fe8de243640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:04.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.851+0000 7fe8de243640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d80705f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:04.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.851+0000 7fe8de243640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d80709d0 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:04.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.851+0000 7fe8de243640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d8140a40 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:04.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.851+0000 7fe8de243640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d81327a0 con 0x7fe8d8140a40 2024-03-25T19:36:04.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.851+0000 7fe8de243640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d8144280 con 0x7fe8d80709d0 2024-03-25T19:36:04.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.851+0000 7fe8de243640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d8142fd0 con 0x7fe8d80705f0 2024-03-25T19:36:04.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.851+0000 7fe8de243640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:04.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.851+0000 7fe8de243640 10 monclient(hunting): authenticate will time out at 6418.121273s 2024-03-25T19:36:04.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.856+0000 7fe8d77fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fe8d80705f0 0x7fe8d80a98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:36400/0 (socket says 172.21.15.19:36400) 2024-03-25T19:36:04.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.856+0000 7fe8d77fe640 1 -- 172.21.15.19:0/996703128 learned_addr learned my addr 172.21.15.19:0/996703128 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:04.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.857+0000 7fe8bffff640 10 client.?.objecter ms_handle_connect 0x7fe8d80705f0 2024-03-25T19:36:04.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.857+0000 7fe8bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:04.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.858+0000 7fe8bffff640 10 client.?.objecter ms_handle_connect 0x7fe8d80709d0 2024-03-25T19:36:04.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.859+0000 7fe8bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:04.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.859+0000 7fe8bffff640 10 client.?.objecter ms_handle_connect 0x7fe8d8140a40 2024-03-25T19:36:04.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.859+0000 7fe8bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:04.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.865+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712364917 0 0) 0x7fe8d81327a0 con 0x7fe8d8140a40 2024-03-25T19:36:04.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.865+0000 7fe8bffff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.865+0000 7fe8bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.865+0000 7fe8bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.865+0000 7fe8bffff640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:36:04.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.865+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c4004eb0 con 0x7fe8d8140a40 2024-03-25T19:36:04.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.874+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847841974 0 0) 0x7fe8d8142fd0 con 0x7fe8d80705f0 2024-03-25T19:36:04.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.875+0000 7fe8bffff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.875+0000 7fe8bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.875+0000 7fe8bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.875+0000 7fe8bffff640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:36:04.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.875+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d81327a0 con 0x7fe8d80705f0 2024-03-25T19:36:04.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.878+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391012930 0 0) 0x7fe8d8144280 con 0x7fe8d80709d0 2024-03-25T19:36:04.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.878+0000 7fe8bffff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.878+0000 7fe8bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.878+0000 7fe8bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.878+0000 7fe8bffff640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:36:04.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.878+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d8142fd0 con 0x7fe8d80709d0 2024-03-25T19:36:04.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.888+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2511871849 0 0) 0x7fe8c4004eb0 con 0x7fe8d8140a40 2024-03-25T19:36:04.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.888+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8d8144280 con 0x7fe8d8140a40 2024-03-25T19:36:04.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.888+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe8d0001e80 con 0x7fe8d8140a40 2024-03-25T19:36:04.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.888+0000 7fe8bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.888+0000 7fe8bffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:04.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.888+0000 7fe8bffff640 10 monclient(hunting): dump: 2024-03-25T19:36:04.889 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:04.889 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:04.889 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:04.889 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:04.889 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.889 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:04.889 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:04.889 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:04.889 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:04.889 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:04.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.888+0000 7fe8bffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:04.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.894+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3027085336 0 0) 0x7fe8d81327a0 con 0x7fe8d80705f0 2024-03-25T19:36:04.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.894+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8c4004eb0 con 0x7fe8d80705f0 2024-03-25T19:36:04.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.894+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe8c80019b0 con 0x7fe8d80705f0 2024-03-25T19:36:04.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.894+0000 7fe8bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.894+0000 7fe8bffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:04.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.894+0000 7fe8bffff640 10 monclient(hunting): dump: 2024-03-25T19:36:04.894 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:04.894 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:04.895 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:04.895 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:04.895 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.895 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:04.895 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:04.895 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:04.895 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:04.895 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:04.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 790751039 0 0) 0x7fe8d8144280 con 0x7fe8d8140a40 2024-03-25T19:36:04.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 >> v1:172.21.15.19:6790/0 conn(0x7fe8d80705f0 legacy=0x7fe8d80a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 >> v1:172.21.15.188:6789/0 conn(0x7fe8d80709d0 legacy=0x7fe8d81428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8bffff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:04.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8bffff640 1 monclient: found mon.a 2024-03-25T19:36:04.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:04.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8bffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:04.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d80ab590 con 0x7fe8d8140a40 2024-03-25T19:36:04.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8bffff640 10 monclient: _finish_auth 0 2024-03-25T19:36:04.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8bffff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:04.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:04.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8de243640 5 monclient: authenticate success, global_id 4548 2024-03-25T19:36:04.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8de243640 10 monclient: _renew_subs 2024-03-25T19:36:04.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8de243640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:04.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8de243640 1 -- 172.21.15.19:0/996703128 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d80ab870 con 0x7fe8d8140a40 2024-03-25T19:36:04.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8de243640 10 client.4548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:04.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8de243640 10 monclient: _renew_subs 2024-03-25T19:36:04.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8de243640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:04.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8de243640 1 -- 172.21.15.19:0/996703128 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d80af590 con 0x7fe8d8140a40 2024-03-25T19:36:04.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.904+0000 7fe8de243640 1 librados: init done 2024-03-25T19:36:04.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.905+0000 7fe8de243640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:04.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.905+0000 7fe8de243640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:04.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.905+0000 7fe8de243640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:04.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:04.905+0000 7fe8de243640 1 -- 172.21.15.19:0/996703128 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d80627b0 con 0x7fe8d8140a40 2024-03-25T19:36:05.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.297+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8d0002020 con 0x7fe8d8140a40 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.297+0000 7fe8bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.297+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe8d0004fb0 con 0x7fe8d8140a40 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.297+0000 7fe8bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.297+0000 7fe8bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.297+0000 7fe8bffff640 10 monclient: dump: 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:05.298 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:05.299 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:05.299 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:05.299 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:05.299 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:05.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.297+0000 7fe8d5ffb640 4 set_mon_vals no callback set 2024-03-25T19:36:05.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.298+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fe8d001af80 con 0x7fe8d8140a40 2024-03-25T19:36:05.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.298+0000 7fe8d77fe640 10 monclient: get_auth_request con 0x7fe8c406df20 auth_method 0 2024-03-25T19:36:05.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.299+0000 7fe8bffff640 10 client.4548.objecter ms_handle_connect 0x7fe8c406df20 2024-03-25T19:36:05.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.304+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fe8d0054aa0 con 0x7fe8d8140a40 2024-03-25T19:36:05.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.304+0000 7fe8bffff640 10 client.4548.objecter ms_dispatch 0x7fe8d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:05.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.304+0000 7fe8bffff640 3 client.4548.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:05.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.304+0000 7fe8bffff640 3 client.4548.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:05.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.304+0000 7fe8bffff640 20 client.4548.objecter dump_active .. 0 homeless 2024-03-25T19:36:05.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.304+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8d00503e0 con 0x7fe8d8140a40 2024-03-25T19:36:05.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.304+0000 7fe8bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:05.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.304+0000 7fe8bffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:05.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.561+0000 7fe8de243640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:05.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.561+0000 7fe8de243640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:05.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.561+0000 7fe8de243640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:05.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.561+0000 7fe8de243640 1 -- 172.21.15.19:0/996703128 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8d8144430 con 0x7fe8d8140a40 2024-03-25T19:36:05.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.587+0000 7fe8bffff640 1 -- 172.21.15.19:0/996703128 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fe8d8144430 con 0x7fe8d8140a40 2024-03-25T19:36:05.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.587+0000 7fe8bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:05.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.587+0000 7fe8bffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:05.587 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:36:05.587 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:05.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 10 librados: watch_flush enter 2024-03-25T19:36:05.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 10 librados: watch_flush exit 2024-03-25T19:36:05.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 20 client.4548.objecter shutdown clearing up homeless session... 2024-03-25T19:36:05.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 10 client.4548.objecter successfully canceled tick 2024-03-25T19:36:05.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 1 -- 172.21.15.19:0/996703128 >> v1:172.21.15.188:6800/2207506229 conn(0x7fe8c406df20 legacy=0x7fe8c40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:05.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 10 monclient: shutdown 2024-03-25T19:36:05.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:05.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 1 -- 172.21.15.19:0/996703128 >> v1:172.21.15.19:6789/0 conn(0x7fe8d8140a40 legacy=0x7fe8d81e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:05.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 1 -- 172.21.15.19:0/996703128 shutdown_connections 2024-03-25T19:36:05.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 1 -- 172.21.15.19:0/996703128 >> 172.21.15.19:0/996703128 conn(0x7fe8d80b5c00 msgr2=0x7fe8d80b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:05.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 1 -- 172.21.15.19:0/996703128 shutdown_connections 2024-03-25T19:36:05.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 1 -- 172.21.15.19:0/996703128 wait complete. 2024-03-25T19:36:05.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:05.589+0000 7fe8de243640 1 librados: shutdown 2024-03-25T19:36:05.601 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:12.407 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:12.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.526+0000 7f57bea9a640 10 monclient: get_monmap_and_config 2024-03-25T19:36:12.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.526+0000 7f57bea9a640 10 monclient: build_initial_monmap 2024-03-25T19:36:12.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.526+0000 7f57bea9a640 10 monclient: monmap: 2024-03-25T19:36:12.526 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:12.526 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:12.526 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:36:12.527293+0000 2024-03-25T19:36:12.526 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:36:12.527293+0000 2024-03-25T19:36:12.526 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:12.526 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:12.526 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:12.526 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:36:12.527 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:36:12.527 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:12.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.526+0000 7f57bea9a640 1 Processor -- start 2024-03-25T19:36:12.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.527+0000 7f57bea9a640 1 -- start start 2024-03-25T19:36:12.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.527+0000 7f57bea9a640 10 monclient: init 2024-03-25T19:36:12.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.527+0000 7f57bea9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:12.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.527+0000 7f57bea9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:12.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.527+0000 7f57bea9a640 10 monclient(hunting): picked mon.noname-a con 0x7f57b80a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:12.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.527+0000 7f57bea9a640 10 monclient(hunting): picked mon.noname-b con 0x7f57b813f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:12.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.527+0000 7f57bea9a640 10 monclient(hunting): picked mon.noname-c con 0x7f57b81430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:12.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.527+0000 7f57bea9a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b8142520 con 0x7f57b80a98f0 2024-03-25T19:36:12.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.527+0000 7f57bea9a640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b80b28a0 con 0x7f57b81430c0 2024-03-25T19:36:12.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.527+0000 7f57bea9a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b80b2a30 con 0x7f57b813f710 2024-03-25T19:36:12.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.527+0000 7f57bea9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:12.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.527+0000 7f57bea9a640 10 monclient(hunting): authenticate will time out at 6425.797843s 2024-03-25T19:36:12.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.530+0000 7f57bd037640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f57b81430c0 0x7f57b814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:55312/0 (socket says 172.21.15.19:55312) 2024-03-25T19:36:12.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.530+0000 7f57bd037640 1 -- 172.21.15.19:0/1807289088 learned_addr learned my addr 172.21.15.19:0/1807289088 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:12.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.540+0000 7f57b77fe640 1 -- 172.21.15.19:0/1807289088 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945637643 0 0) 0x7f57b80b28a0 con 0x7f57b81430c0 2024-03-25T19:36:12.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.540+0000 7f57b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:12.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.540+0000 7f57b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:12.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.540+0000 7f57b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:12.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.540+0000 7f57b77fe640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:36:12.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.540+0000 7f57b77fe640 1 -- 172.21.15.19:0/1807289088 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a8003520 con 0x7f57b81430c0 2024-03-25T19:36:12.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.543+0000 7f57b77fe640 1 -- 172.21.15.19:0/1807289088 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008139458 0 0) 0x7f57b80b2a30 con 0x7f57b813f710 2024-03-25T19:36:12.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.543+0000 7f57b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:12.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.543+0000 7f57b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:12.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.543+0000 7f57b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:12.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.543+0000 7f57b77fe640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:36:12.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.543+0000 7f57b77fe640 1 -- 172.21.15.19:0/1807289088 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b80b28a0 con 0x7f57b813f710 2024-03-25T19:36:12.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 1 -- 172.21.15.19:0/1807289088 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4039045945 0 0) 0x7f57b80b28a0 con 0x7f57b813f710 2024-03-25T19:36:12.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 1 -- 172.21.15.19:0/1807289088 >> v1:172.21.15.188:6789/0 conn(0x7f57b81430c0 legacy=0x7f57b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 1 -- 172.21.15.19:0/1807289088 >> v1:172.21.15.19:6789/0 conn(0x7f57b80a98f0 legacy=0x7f57b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:36:12.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:12.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 1 monclient: found mon.noname-b 2024-03-25T19:36:12.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:12.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:36:12.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 1 -- 172.21.15.19:0/1807289088 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b80b44c0 con 0x7f57b813f710 2024-03-25T19:36:12.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:12.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 1 -- 172.21.15.19:0/1807289088 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f57a4001670 con 0x7f57b813f710 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57bea9a640 5 monclient: authenticate success, global_id 4517 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57bea9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.549+0000 7f57b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.550+0000 7f57b77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.550+0000 7f57b77fe640 10 monclient: dump: 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:12.551 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:12.552 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:12.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.550+0000 7f57bea9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:12.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.559+0000 7f57b77fe640 1 -- 172.21.15.19:0/1807289088 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57a40017d0 con 0x7f57b813f710 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.559+0000 7f57b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.559+0000 7f57b77fe640 1 -- 172.21.15.19:0/1807289088 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f57a4001c20 con 0x7f57b813f710 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.559+0000 7f57bea9a640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.559+0000 7f57bea9a640 4 set_mon_vals no callback set 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.559+0000 7f57b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.559+0000 7f57b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.559+0000 7f57b77fe640 10 monclient: dump: 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.561 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:12.562 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:12.562 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:12.562 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:12.562 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:12.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.560+0000 7f57bea9a640 10 monclient: shutdown 2024-03-25T19:36:12.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.560+0000 7f57bea9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:12.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.560+0000 7f57bea9a640 1 -- 172.21.15.19:0/1807289088 >> v1:172.21.15.19:6790/0 conn(0x7f57b813f710 legacy=0x7f57b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.560+0000 7f57bea9a640 1 -- 172.21.15.19:0/1807289088 shutdown_connections 2024-03-25T19:36:12.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.560+0000 7f57bea9a640 1 -- 172.21.15.19:0/1807289088 >> 172.21.15.19:0/1807289088 conn(0x7f57b80b7ea0 msgr2=0x7f57b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:12.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.560+0000 7f57bea9a640 1 -- 172.21.15.19:0/1807289088 shutdown_connections 2024-03-25T19:36:12.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.560+0000 7f57bea9a640 1 -- 172.21.15.19:0/1807289088 wait complete. 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.561+0000 7f57bea9a640 10 monclient: build_initial_monmap 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.561+0000 7f57bea9a640 10 monclient: monmap: 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.562+0000 7f57bea9a640 1 librados: starting msgr at 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.562+0000 7f57bea9a640 1 librados: starting objecter 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.562+0000 7f57bea9a640 1 Processor -- start 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.562+0000 7f57bea9a640 1 -- start start 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.562+0000 7f57bea9a640 1 librados: setting wanted keys 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.562+0000 7f57bea9a640 1 librados: calling monclient init 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.562+0000 7f57bea9a640 10 monclient: init 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.563+0000 7f57bea9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.563+0000 7f57bea9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.563+0000 7f57bea9a640 10 monclient(hunting): picked mon.noname-a con 0x7f57b80a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.563+0000 7f57bea9a640 10 monclient(hunting): picked mon.noname-c con 0x7f57b813f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.563+0000 7f57bea9a640 10 monclient(hunting): picked mon.noname-b con 0x7f57b81430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.563+0000 7f57bea9a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b80b06b0 con 0x7f57b80a98f0 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.563+0000 7f57bea9a640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b81d80f0 con 0x7f57b81430c0 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.563+0000 7f57bea9a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b80b08e0 con 0x7f57b813f710 2024-03-25T19:36:12.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.563+0000 7f57bea9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:12.566 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.563+0000 7f57bea9a640 10 monclient(hunting): authenticate will time out at 6425.833318s 2024-03-25T19:36:12.572 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.572+0000 7f57b7fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f57b813f710 0x7f57b80b0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:60580/0 (socket says 172.21.15.19:60580) 2024-03-25T19:36:12.572 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.572+0000 7f57b7fff640 1 -- 172.21.15.19:0/1846365963 learned_addr learned my addr 172.21.15.19:0/1846365963 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:12.574 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.574+0000 7f57b57fa640 10 client.?.objecter ms_handle_connect 0x7f57b813f710 2024-03-25T19:36:12.574 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.574+0000 7f57b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:12.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.578+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206916513 0 0) 0x7f57b80b08e0 con 0x7f57b813f710 2024-03-25T19:36:12.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.578+0000 7f57b57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:12.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.578+0000 7f57b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:12.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.578+0000 7f57b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:12.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.578+0000 7f57b57fa640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:36:12.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.578+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f579c004eb0 con 0x7f57b813f710 2024-03-25T19:36:12.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.583+0000 7f57b57fa640 10 client.?.objecter ms_handle_connect 0x7f57b81430c0 2024-03-25T19:36:12.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.583+0000 7f57b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:12.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.589+0000 7f57b57fa640 10 client.?.objecter ms_handle_connect 0x7f57b80a98f0 2024-03-25T19:36:12.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.589+0000 7f57b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:12.594 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.594+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684236236 0 0) 0x7f57b81d80f0 con 0x7f57b81430c0 2024-03-25T19:36:12.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.594+0000 7f57b57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:12.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.594+0000 7f57b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:12.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.594+0000 7f57b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:12.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.594+0000 7f57b57fa640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:36:12.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.594+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b80b08e0 con 0x7f57b81430c0 2024-03-25T19:36:12.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.599+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3218017742 0 0) 0x7f579c004eb0 con 0x7f57b813f710 2024-03-25T19:36:12.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.599+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57b81d80f0 con 0x7f57b813f710 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.599+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f57a4001630 con 0x7f57b813f710 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.599+0000 7f57b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.599+0000 7f57b57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.599+0000 7f57b57fa640 10 monclient(hunting): dump: 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:12.600 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:12.601 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:12.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.599+0000 7f57b57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:12.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.602+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377445411 0 0) 0x7f57b80b06b0 con 0x7f57b80a98f0 2024-03-25T19:36:12.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.602+0000 7f57b57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:12.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.602+0000 7f57b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:12.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.602+0000 7f57b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:12.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.602+0000 7f57b57fa640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:36:12.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.602+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f579c004eb0 con 0x7f57b80a98f0 2024-03-25T19:36:12.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 646182944 0 0) 0x7f57b81d80f0 con 0x7f57b813f710 2024-03-25T19:36:12.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 >> v1:172.21.15.188:6789/0 conn(0x7f57b81430c0 legacy=0x7f57b80acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 >> v1:172.21.15.19:6789/0 conn(0x7f57b80a98f0 legacy=0x7f57b81d7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57b57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:36:12.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57b57fa640 1 monclient: found mon.c 2024-03-25T19:36:12.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57b57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:12.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57b57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:36:12.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b80ad420 con 0x7f57b813f710 2024-03-25T19:36:12.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57b57fa640 10 monclient: _finish_auth 0 2024-03-25T19:36:12.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57b57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:36:12.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:12.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57bea9a640 5 monclient: authenticate success, global_id 4520 2024-03-25T19:36:12.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57bea9a640 10 monclient: _renew_subs 2024-03-25T19:36:12.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57bea9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:36:12.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57bea9a640 1 -- 172.21.15.19:0/1846365963 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b80adf30 con 0x7f57b813f710 2024-03-25T19:36:12.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57bea9a640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:12.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57bea9a640 10 monclient: _renew_subs 2024-03-25T19:36:12.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57bea9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:36:12.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57bea9a640 1 -- 172.21.15.19:0/1846365963 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b80ae910 con 0x7f57b813f710 2024-03-25T19:36:12.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.606+0000 7f57bea9a640 1 librados: init done 2024-03-25T19:36:12.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.607+0000 7f57bea9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:12.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.607+0000 7f57bea9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:12.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.607+0000 7f57bea9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:36:12.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.607+0000 7f57bea9a640 1 -- 172.21.15.19:0/1846365963 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b80b1710 con 0x7f57b813f710 2024-03-25T19:36:12.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.632+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57a40038a0 con 0x7f57b813f710 2024-03-25T19:36:12.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.632+0000 7f57b57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:12.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.632+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f57a4003ad0 con 0x7f57b813f710 2024-03-25T19:36:12.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.632+0000 7f57b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.632+0000 7f57b57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:12.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.632+0000 7f57b57fa640 10 monclient: dump: 2024-03-25T19:36:12.632 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:12.632 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:12.632 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:12.632 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:12.632 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.633 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:12.633 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:12.633 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:12.633 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:12.633 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:12.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.632+0000 7f57b6ffd640 4 set_mon_vals no callback set 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.633+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f57a4003cf0 con 0x7f57b813f710 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.633+0000 7f57bc836640 10 monclient: get_auth_request con 0x7f579c06de80 auth_method 0 2024-03-25T19:36:12.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.633+0000 7f57b57fa640 10 client.4520.objecter ms_handle_connect 0x7f579c06de80 2024-03-25T19:36:12.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.635+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f57a4054810 con 0x7f57b813f710 2024-03-25T19:36:12.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.635+0000 7f57b57fa640 10 client.4520.objecter ms_dispatch 0x7f57b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:12.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.635+0000 7f57b57fa640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:12.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.635+0000 7f57b57fa640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:12.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.635+0000 7f57b57fa640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-25T19:36:12.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.936+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57a4050320 con 0x7f57b813f710 2024-03-25T19:36:12.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.937+0000 7f57b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:12.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:12.937+0000 7f57b57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:13.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.169+0000 7f57bea9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:13.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.169+0000 7f57bea9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:13.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.169+0000 7f57bea9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:36:13.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.169+0000 7f57bea9a640 1 -- 172.21.15.19:0/1846365963 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57b814f2d0 con 0x7f57b813f710 2024-03-25T19:36:13.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.190+0000 7f57b57fa640 1 -- 172.21.15.19:0/1846365963 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f57a4052250 con 0x7f57b813f710 2024-03-25T19:36:13.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.190+0000 7f57b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:13.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.190+0000 7f57b57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:13.190 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:36:13.190 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:13.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.192+0000 7f57bea9a640 10 librados: watch_flush enter 2024-03-25T19:36:13.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.192+0000 7f57bea9a640 10 librados: watch_flush exit 2024-03-25T19:36:13.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.192+0000 7f57bea9a640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-25T19:36:13.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.192+0000 7f57bea9a640 10 client.4520.objecter successfully canceled tick 2024-03-25T19:36:13.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.192+0000 7f57bea9a640 1 -- 172.21.15.19:0/1846365963 >> v1:172.21.15.188:6800/2207506229 conn(0x7f579c06de80 legacy=0x7f579c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:13.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.192+0000 7f57bea9a640 10 monclient: shutdown 2024-03-25T19:36:13.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.192+0000 7f57bea9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:13.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.192+0000 7f57bea9a640 1 -- 172.21.15.19:0/1846365963 >> v1:172.21.15.19:6790/0 conn(0x7f57b813f710 legacy=0x7f57b80b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:13.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.192+0000 7f57bea9a640 1 -- 172.21.15.19:0/1846365963 shutdown_connections 2024-03-25T19:36:13.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.192+0000 7f57bea9a640 1 -- 172.21.15.19:0/1846365963 >> 172.21.15.19:0/1846365963 conn(0x7f57b80b7ea0 msgr2=0x7f57b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:13.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.193+0000 7f57bea9a640 1 -- 172.21.15.19:0/1846365963 shutdown_connections 2024-03-25T19:36:13.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.193+0000 7f57bea9a640 1 -- 172.21.15.19:0/1846365963 wait complete. 2024-03-25T19:36:13.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:13.193+0000 7f57bea9a640 1 librados: shutdown 2024-03-25T19:36:13.205 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:18.007 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:07:40.710 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:18.008 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:07:40.737 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:18.008 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:07:40.711 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:18.015 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:07:40.745 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:18.019 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:07:40.722 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:18.049 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:07:40.779 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:18.081 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:07:40.811 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:18.100 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:07:40.830 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:18.133 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:07:40.862 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:20.012 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:20.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.114+0000 7feab7e0d640 10 monclient: get_monmap_and_config 2024-03-25T19:36:20.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.114+0000 7feab7e0d640 10 monclient: build_initial_monmap 2024-03-25T19:36:20.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.114+0000 7feab7e0d640 10 monclient: monmap: 2024-03-25T19:36:20.114 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:20.114 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:20.114 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:36:20.115402+0000 2024-03-25T19:36:20.115 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:36:20.115402+0000 2024-03-25T19:36:20.115 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:20.115 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:20.115 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:20.115 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:36:20.115 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:36:20.115 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:20.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.114+0000 7feab7e0d640 1 Processor -- start 2024-03-25T19:36:20.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.115+0000 7feab7e0d640 1 -- start start 2024-03-25T19:36:20.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.115+0000 7feab7e0d640 10 monclient: init 2024-03-25T19:36:20.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.115+0000 7feab7e0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:20.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.115+0000 7feab7e0d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:20.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.115+0000 7feab7e0d640 10 monclient(hunting): picked mon.noname-c con 0x7feab00a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:20.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.115+0000 7feab7e0d640 10 monclient(hunting): picked mon.noname-a con 0x7feab013f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:20.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.115+0000 7feab7e0d640 10 monclient(hunting): picked mon.noname-b con 0x7feab01430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:20.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.116+0000 7feab7e0d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab0142520 con 0x7feab013f710 2024-03-25T19:36:20.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.116+0000 7feab7e0d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab00b28a0 con 0x7feab00a98f0 2024-03-25T19:36:20.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.116+0000 7feab7e0d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab00b2a30 con 0x7feab01430c0 2024-03-25T19:36:20.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.116+0000 7feab7e0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:20.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.116+0000 7feab7e0d640 10 monclient(hunting): authenticate will time out at 6433.386025s 2024-03-25T19:36:20.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.122+0000 7feab53a8640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7feab013f710 0x7feab01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:35054/0 (socket says 172.21.15.19:35054) 2024-03-25T19:36:20.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.122+0000 7feab53a8640 1 -- 172.21.15.19:0/783600542 learned_addr learned my addr 172.21.15.19:0/783600542 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:20.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.130+0000 7feab4ba7640 1 -- 172.21.15.19:0/783600542 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923117434 0 0) 0x7feab0142520 con 0x7feab013f710 2024-03-25T19:36:20.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.131+0000 7feab4ba7640 10 monclient(hunting): _negotiate 2024-03-25T19:36:20.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.131+0000 7feab4ba7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:20.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.131+0000 7feab4ba7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:20.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.131+0000 7feab4ba7640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:36:20.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.131+0000 7feab4ba7640 1 -- 172.21.15.19:0/783600542 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea9c003520 con 0x7feab013f710 2024-03-25T19:36:20.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.142+0000 7feab4ba7640 1 -- 172.21.15.19:0/783600542 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637355058 0 0) 0x7feab00b28a0 con 0x7feab00a98f0 2024-03-25T19:36:20.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.142+0000 7feab4ba7640 10 monclient(hunting): _negotiate 2024-03-25T19:36:20.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.142+0000 7feab4ba7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:20.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.142+0000 7feab4ba7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:20.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.142+0000 7feab4ba7640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:36:20.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.142+0000 7feab4ba7640 1 -- 172.21.15.19:0/783600542 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab0142520 con 0x7feab00a98f0 2024-03-25T19:36:20.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 1 -- 172.21.15.19:0/783600542 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1314645039 0 0) 0x7fea9c003520 con 0x7feab013f710 2024-03-25T19:36:20.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 1 -- 172.21.15.19:0/783600542 >> v1:172.21.15.19:6790/0 conn(0x7feab01430c0 legacy=0x7feab014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 1 -- 172.21.15.19:0/783600542 >> v1:172.21.15.188:6789/0 conn(0x7feab00a98f0 legacy=0x7feab00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 10 monclient: _finish_hunting 0 2024-03-25T19:36:20.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 1 monclient: found mon.noname-a 2024-03-25T19:36:20.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:20.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:36:20.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 1 -- 172.21.15.19:0/783600542 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab00b44c0 con 0x7feab013f710 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 10 monclient: _finish_auth 0 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 10 monclient: _check_auth_tickets 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 1 -- 172.21.15.19:0/783600542 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7feaa0001690 con 0x7feab013f710 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab4ba7640 10 monclient: dump: 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:20.145 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:20.146 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:20.146 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:20.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab7e0d640 5 monclient: authenticate success, global_id 4554 2024-03-25T19:36:20.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.144+0000 7feab7e0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:20.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.148+0000 7feab4ba7640 1 -- 172.21.15.19:0/783600542 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feaa00017f0 con 0x7feab013f710 2024-03-25T19:36:20.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.148+0000 7feab4ba7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:20.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.148+0000 7feab4ba7640 1 -- 172.21.15.19:0/783600542 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7feaa0001c00 con 0x7feab013f710 2024-03-25T19:36:20.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.148+0000 7feab7e0d640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:20.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.148+0000 7feab7e0d640 4 set_mon_vals no callback set 2024-03-25T19:36:20.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.148+0000 7feab4ba7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:20.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.148+0000 7feab4ba7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:20.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.148+0000 7feab4ba7640 10 monclient: dump: 2024-03-25T19:36:20.148 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:20.148 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:20.148 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:20.148 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:20.149 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:20.149 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:20.149 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:20.149 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:20.149 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:20.149 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:20.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.148+0000 7feab7e0d640 10 monclient: shutdown 2024-03-25T19:36:20.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.148+0000 7feab7e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:20.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.148+0000 7feab7e0d640 1 -- 172.21.15.19:0/783600542 >> v1:172.21.15.19:6789/0 conn(0x7feab013f710 legacy=0x7feab01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.149+0000 7feab7e0d640 1 -- 172.21.15.19:0/783600542 shutdown_connections 2024-03-25T19:36:20.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.149+0000 7feab7e0d640 1 -- 172.21.15.19:0/783600542 >> 172.21.15.19:0/783600542 conn(0x7feab00b7ea0 msgr2=0x7feab00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.149+0000 7feab7e0d640 1 -- 172.21.15.19:0/783600542 shutdown_connections 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.149+0000 7feab7e0d640 1 -- 172.21.15.19:0/783600542 wait complete. 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.150+0000 7feab7e0d640 10 monclient: build_initial_monmap 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.150+0000 7feab7e0d640 10 monclient: monmap: 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:20.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.150+0000 7feab7e0d640 1 librados: starting msgr at 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.150+0000 7feab7e0d640 1 librados: starting objecter 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.150+0000 7feab7e0d640 1 Processor -- start 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.150+0000 7feab7e0d640 1 -- start start 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.150+0000 7feab7e0d640 1 librados: setting wanted keys 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.150+0000 7feab7e0d640 1 librados: calling monclient init 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.150+0000 7feab7e0d640 10 monclient: init 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.151+0000 7feab7e0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.151+0000 7feab7e0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.151+0000 7feab7e0d640 10 monclient(hunting): picked mon.noname-b con 0x7feab00a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.151+0000 7feab7e0d640 10 monclient(hunting): picked mon.noname-c con 0x7feab013f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.151+0000 7feab7e0d640 10 monclient(hunting): picked mon.noname-a con 0x7feab01430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.151+0000 7feab7e0d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab00b79f0 con 0x7feab01430c0 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.151+0000 7feab7e0d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab01350e0 con 0x7feab00a98f0 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.151+0000 7feab7e0d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab0135270 con 0x7feab013f710 2024-03-25T19:36:20.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.151+0000 7feab7e0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:20.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.151+0000 7feab7e0d640 10 monclient(hunting): authenticate will time out at 6433.421166s 2024-03-25T19:36:20.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.154+0000 7feab63aa640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7feab01430c0 0x7feab01e20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:35068/0 (socket says 172.21.15.19:35068) 2024-03-25T19:36:20.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.154+0000 7feab63aa640 1 -- 172.21.15.19:0/1071383937 learned_addr learned my addr 172.21.15.19:0/1071383937 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:20.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.154+0000 7feaa67fc640 10 client.?.objecter ms_handle_connect 0x7feab01430c0 2024-03-25T19:36:20.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.154+0000 7feaa67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:20.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.163+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344965921 0 0) 0x7feab00b79f0 con 0x7feab01430c0 2024-03-25T19:36:20.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.163+0000 7feaa67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:20.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.163+0000 7feaa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:20.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.163+0000 7feaa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:20.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.163+0000 7feaa67fc640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:36:20.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.163+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea94004eb0 con 0x7feab01430c0 2024-03-25T19:36:20.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.167+0000 7feaa67fc640 10 client.?.objecter ms_handle_connect 0x7feab00a98f0 2024-03-25T19:36:20.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.167+0000 7feaa67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:20.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.168+0000 7feaa67fc640 10 client.?.objecter ms_handle_connect 0x7feab013f710 2024-03-25T19:36:20.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.168+0000 7feaa67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:20.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.175+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 692519040 0 0) 0x7fea94004eb0 con 0x7feab01430c0 2024-03-25T19:36:20.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.176+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feab00b79f0 con 0x7feab01430c0 2024-03-25T19:36:20.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.176+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7feaa8001c90 con 0x7feab01430c0 2024-03-25T19:36:20.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.176+0000 7feaa67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:20.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.176+0000 7feaa67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:20.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.176+0000 7feaa67fc640 10 monclient(hunting): dump: 2024-03-25T19:36:20.176 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.176+0000 7feaa67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.176+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060594862 0 0) 0x7feab01350e0 con 0x7feab00a98f0 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.176+0000 7feaa67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.176+0000 7feaa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:20.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.176+0000 7feaa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:20.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.176+0000 7feaa67fc640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:36:20.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.176+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea94004eb0 con 0x7feab00a98f0 2024-03-25T19:36:20.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.188+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320563402 0 0) 0x7feab0135270 con 0x7feab013f710 2024-03-25T19:36:20.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.188+0000 7feaa67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:20.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.188+0000 7feaa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:20.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.188+0000 7feaa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:20.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.188+0000 7feaa67fc640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:36:20.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.188+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab01350e0 con 0x7feab013f710 2024-03-25T19:36:20.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1470739975 0 0) 0x7feab00b79f0 con 0x7feab01430c0 2024-03-25T19:36:20.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 >> v1:172.21.15.19:6790/0 conn(0x7feab013f710 legacy=0x7feab01d7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 >> v1:172.21.15.188:6789/0 conn(0x7feab00a98f0 legacy=0x7feab01d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feaa67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feaa67fc640 1 monclient: found mon.a 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feaa67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feaa67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab0135400 con 0x7feab01430c0 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feaa67fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feaa67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feaa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feab7e0d640 5 monclient: authenticate success, global_id 4557 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feab7e0d640 10 monclient: _renew_subs 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feab7e0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.193+0000 7feab7e0d640 1 -- 172.21.15.19:0/1071383937 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab013b630 con 0x7feab01430c0 2024-03-25T19:36:20.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.194+0000 7feab7e0d640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:20.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.194+0000 7feab7e0d640 10 monclient: _renew_subs 2024-03-25T19:36:20.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.194+0000 7feab7e0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:20.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.194+0000 7feab7e0d640 1 -- 172.21.15.19:0/1071383937 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feab0135b10 con 0x7feab01430c0 2024-03-25T19:36:20.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.194+0000 7feab7e0d640 1 librados: init done 2024-03-25T19:36:20.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.195+0000 7feab7e0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:20.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.195+0000 7feab7e0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:20.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.195+0000 7feab7e0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:20.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.195+0000 7feab7e0d640 1 -- 172.21.15.19:0/1071383937 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab00627b0 con 0x7feab01430c0 2024-03-25T19:36:20.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.506+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feaa8003990 con 0x7feab01430c0 2024-03-25T19:36:20.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.506+0000 7feaa67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:20.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.506+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7feaa8003c00 con 0x7feab01430c0 2024-03-25T19:36:20.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.506+0000 7feaa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:20.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.506+0000 7feaa67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:20.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.506+0000 7feaa67fc640 10 monclient: dump: 2024-03-25T19:36:20.507 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:20.507 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:20.507 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:20.507 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:20.507 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:20.507 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:20.507 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:20.507 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:20.508 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:20.508 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:20.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.506+0000 7feaa7fff640 4 set_mon_vals no callback set 2024-03-25T19:36:20.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.507+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7feaa8003e20 con 0x7feab01430c0 2024-03-25T19:36:20.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.508+0000 7feab5ba9640 10 monclient: get_auth_request con 0x7fea9406de60 auth_method 0 2024-03-25T19:36:20.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.508+0000 7feaa67fc640 10 client.4557.objecter ms_handle_connect 0x7fea9406de60 2024-03-25T19:36:20.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.514+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7feaa8054990 con 0x7feab01430c0 2024-03-25T19:36:20.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.514+0000 7feaa67fc640 10 client.4557.objecter ms_dispatch 0x7feab0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:20.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.514+0000 7feaa67fc640 3 client.4557.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:20.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.514+0000 7feaa67fc640 3 client.4557.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:20.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.514+0000 7feaa67fc640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-25T19:36:20.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.514+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feaa80504a0 con 0x7feab01430c0 2024-03-25T19:36:20.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.514+0000 7feaa67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:20.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.514+0000 7feaa67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:20.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.774+0000 7feab7e0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:20.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.774+0000 7feab7e0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:20.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.774+0000 7feab7e0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:20.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.774+0000 7feab7e0d640 1 -- 172.21.15.19:0/1071383937 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feab013b7c0 con 0x7feab01430c0 2024-03-25T19:36:20.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.804+0000 7feaa67fc640 1 -- 172.21.15.19:0/1071383937 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7feaa8004820 con 0x7feab01430c0 2024-03-25T19:36:20.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.804+0000 7feaa67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:20.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.804+0000 7feaa67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:20.805 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:36:20.805 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:20.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 10 librados: watch_flush enter 2024-03-25T19:36:20.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 10 librados: watch_flush exit 2024-03-25T19:36:20.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-25T19:36:20.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 10 client.4557.objecter successfully canceled tick 2024-03-25T19:36:20.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 1 -- 172.21.15.19:0/1071383937 >> v1:172.21.15.188:6800/2207506229 conn(0x7fea9406de60 legacy=0x7fea94070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 10 monclient: shutdown 2024-03-25T19:36:20.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:20.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 1 -- 172.21.15.19:0/1071383937 >> v1:172.21.15.19:6789/0 conn(0x7feab01430c0 legacy=0x7feab01e20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 1 -- 172.21.15.19:0/1071383937 shutdown_connections 2024-03-25T19:36:20.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 1 -- 172.21.15.19:0/1071383937 >> 172.21.15.19:0/1071383937 conn(0x7feab00b7ea0 msgr2=0x7feab013ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:20.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 1 -- 172.21.15.19:0/1071383937 shutdown_connections 2024-03-25T19:36:20.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 1 -- 172.21.15.19:0/1071383937 wait complete. 2024-03-25T19:36:20.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:20.807+0000 7feab7e0d640 1 librados: shutdown 2024-03-25T19:36:20.819 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:27.625 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.744+0000 7fd9de915640 10 monclient: get_monmap_and_config 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.744+0000 7fd9de915640 10 monclient: build_initial_monmap 2024-03-25T19:36:27.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.744+0000 7fd9de915640 10 monclient: monmap: 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:36:27.745965+0000 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:36:27.745965+0000 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:36:27.745 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:27.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.745+0000 7fd9de915640 1 Processor -- start 2024-03-25T19:36:27.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.745+0000 7fd9de915640 1 -- start start 2024-03-25T19:36:27.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.745+0000 7fd9de915640 10 monclient: init 2024-03-25T19:36:27.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.746+0000 7fd9de915640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:27.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.746+0000 7fd9de915640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:27.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.746+0000 7fd9de915640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d80a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:27.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.746+0000 7fd9de915640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d813f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:27.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.746+0000 7fd9de915640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d81430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:27.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.746+0000 7fd9de915640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d8142520 con 0x7fd9d80a98f0 2024-03-25T19:36:27.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.746+0000 7fd9de915640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d80b28a0 con 0x7fd9d813f710 2024-03-25T19:36:27.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.746+0000 7fd9de915640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d80b2a30 con 0x7fd9d81430c0 2024-03-25T19:36:27.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.746+0000 7fd9de915640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:27.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.746+0000 7fd9de915640 10 monclient(hunting): authenticate will time out at 6441.016414s 2024-03-25T19:36:27.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.753+0000 7fd9d77fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fd9d813f710 0x7fd9d81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:42706/0 (socket says 172.21.15.19:42706) 2024-03-25T19:36:27.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.753+0000 7fd9d77fe640 1 -- 172.21.15.19:0/2908902601 learned_addr learned my addr 172.21.15.19:0/2908902601 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:27.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.771+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116662609 0 0) 0x7fd9d80b28a0 con 0x7fd9d813f710 2024-03-25T19:36:27.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.771+0000 7fd9d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.771+0000 7fd9d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.771+0000 7fd9d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.771+0000 7fd9d6ffd640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:36:27.771 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.771+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c0003520 con 0x7fd9d813f710 2024-03-25T19:36:27.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.777+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469147861 0 0) 0x7fd9d80b2a30 con 0x7fd9d81430c0 2024-03-25T19:36:27.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.777+0000 7fd9d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.777+0000 7fd9d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.777+0000 7fd9d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.777+0000 7fd9d6ffd640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:36:27.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.777+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d80b28a0 con 0x7fd9d81430c0 2024-03-25T19:36:27.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.785+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320758385 0 0) 0x7fd9d8142520 con 0x7fd9d80a98f0 2024-03-25T19:36:27.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.785+0000 7fd9d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.785+0000 7fd9d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.785+0000 7fd9d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.785+0000 7fd9d6ffd640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:36:27.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.785+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d80b2a30 con 0x7fd9d80a98f0 2024-03-25T19:36:27.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1162827785 0 0) 0x7fd9c0003520 con 0x7fd9d813f710 2024-03-25T19:36:27.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 >> v1:172.21.15.19:6790/0 conn(0x7fd9d81430c0 legacy=0x7fd9d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 >> v1:172.21.15.19:6789/0 conn(0x7fd9d80a98f0 legacy=0x7fd9d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:27.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:36:27.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d80b44c0 con 0x7fd9d813f710 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd9cc001690 con 0x7fd9d813f710 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9d6ffd640 10 monclient: dump: 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:27.789 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:27.790 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:27.790 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:27.790 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:27.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9de915640 5 monclient: authenticate success, global_id 4549 2024-03-25T19:36:27.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.788+0000 7fd9de915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:27.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.796+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9cc0017f0 con 0x7fd9d813f710 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.796+0000 7fd9d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.796+0000 7fd9d6ffd640 1 -- 172.21.15.19:0/2908902601 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd9cc001c00 con 0x7fd9d813f710 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.796+0000 7fd9d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.796+0000 7fd9d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.796+0000 7fd9d6ffd640 10 monclient: dump: 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:27.796 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:27.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.796+0000 7fd9de915640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:27.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.796+0000 7fd9de915640 4 set_mon_vals no callback set 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.796+0000 7fd9de915640 10 monclient: shutdown 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.796+0000 7fd9de915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.796+0000 7fd9de915640 1 -- 172.21.15.19:0/2908902601 >> v1:172.21.15.188:6789/0 conn(0x7fd9d813f710 legacy=0x7fd9d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.797+0000 7fd9de915640 1 -- 172.21.15.19:0/2908902601 shutdown_connections 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.797+0000 7fd9de915640 1 -- 172.21.15.19:0/2908902601 >> 172.21.15.19:0/2908902601 conn(0x7fd9d80b7ea0 msgr2=0x7fd9d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.797+0000 7fd9de915640 1 -- 172.21.15.19:0/2908902601 shutdown_connections 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.797+0000 7fd9de915640 1 -- 172.21.15.19:0/2908902601 wait complete. 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.797+0000 7fd9de915640 10 monclient: build_initial_monmap 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.797+0000 7fd9de915640 10 monclient: monmap: 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:36:27.798 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:27.799 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:36:27.799 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:27.799 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:36:27.799 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:36:27.799 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:27.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.798+0000 7fd9de915640 1 librados: starting msgr at 2024-03-25T19:36:27.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.798+0000 7fd9de915640 1 librados: starting objecter 2024-03-25T19:36:27.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.798+0000 7fd9de915640 1 Processor -- start 2024-03-25T19:36:27.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.798+0000 7fd9de915640 1 -- start start 2024-03-25T19:36:27.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.798+0000 7fd9de915640 1 librados: setting wanted keys 2024-03-25T19:36:27.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.798+0000 7fd9de915640 1 librados: calling monclient init 2024-03-25T19:36:27.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.798+0000 7fd9de915640 10 monclient: init 2024-03-25T19:36:27.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.799+0000 7fd9de915640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:27.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.799+0000 7fd9de915640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:27.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.799+0000 7fd9de915640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d80a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:27.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.799+0000 7fd9de915640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d813f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:27.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.799+0000 7fd9de915640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d81430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:27.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.799+0000 7fd9de915640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d8138a70 con 0x7fd9d81430c0 2024-03-25T19:36:27.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.799+0000 7fd9de915640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d80aba50 con 0x7fd9d813f710 2024-03-25T19:36:27.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.799+0000 7fd9de915640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d81d77a0 con 0x7fd9d80a98f0 2024-03-25T19:36:27.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.799+0000 7fd9de915640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:27.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.799+0000 7fd9de915640 10 monclient(hunting): authenticate will time out at 6441.069356s 2024-03-25T19:36:27.818 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.818+0000 7fd9d77fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fd9d813f710 0x7fd9d81d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:42752/0 (socket says 172.21.15.19:42752) 2024-03-25T19:36:27.818 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.818+0000 7fd9d77fe640 1 -- 172.21.15.19:0/1184046333 learned_addr learned my addr 172.21.15.19:0/1184046333 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:27.821 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.822+0000 7fd9d4ff9640 10 client.?.objecter ms_handle_connect 0x7fd9d813f710 2024-03-25T19:36:27.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.822+0000 7fd9d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:27.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.822+0000 7fd9d4ff9640 10 client.?.objecter ms_handle_connect 0x7fd9d81430c0 2024-03-25T19:36:27.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.822+0000 7fd9d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:27.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.830+0000 7fd9d4ff9640 10 client.?.objecter ms_handle_connect 0x7fd9d80a98f0 2024-03-25T19:36:27.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.830+0000 7fd9d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:27.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.837+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422624850 0 0) 0x7fd9d81d77a0 con 0x7fd9d80a98f0 2024-03-25T19:36:27.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.837+0000 7fd9d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.837+0000 7fd9d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.837+0000 7fd9d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.837+0000 7fd9d4ff9640 10 monclient(hunting): my global_id is 4532 2024-03-25T19:36:27.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.837+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c4004eb0 con 0x7fd9d80a98f0 2024-03-25T19:36:27.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.840+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401214207 0 0) 0x7fd9d80aba50 con 0x7fd9d813f710 2024-03-25T19:36:27.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.840+0000 7fd9d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.840+0000 7fd9d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.840+0000 7fd9d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.840+0000 7fd9d4ff9640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:36:27.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.840+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d81d77a0 con 0x7fd9d813f710 2024-03-25T19:36:27.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.847+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098012056 0 0) 0x7fd9d8138a70 con 0x7fd9d81430c0 2024-03-25T19:36:27.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.847+0000 7fd9d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.847+0000 7fd9d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.847+0000 7fd9d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.847+0000 7fd9d4ff9640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:36:27.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.847+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d80aba50 con 0x7fd9d81430c0 2024-03-25T19:36:27.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.866+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3629389607 0 0) 0x7fd9d81d77a0 con 0x7fd9d813f710 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.866+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9d8138a70 con 0x7fd9d813f710 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.866+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd9cc001b40 con 0x7fd9d813f710 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.866+0000 7fd9d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.866+0000 7fd9d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.866+0000 7fd9d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:27.867 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:27.868 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:27.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.866+0000 7fd9d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:27.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.873+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 866972312 0 0) 0x7fd9c4004eb0 con 0x7fd9d80a98f0 2024-03-25T19:36:27.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.873+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9d81d77a0 con 0x7fd9d80a98f0 2024-03-25T19:36:27.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.873+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd9c80018f0 con 0x7fd9d80a98f0 2024-03-25T19:36:27.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.873+0000 7fd9d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.873+0000 7fd9d4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:27.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.873+0000 7fd9d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:27.873 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:27.873 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:27.873 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:27.873 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:27.873 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.873 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:27.874 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:27.874 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:27.874 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:27.874 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1910612144 0 0) 0x7fd9d8138a70 con 0x7fd9d813f710 2024-03-25T19:36:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 >> v1:172.21.15.19:6790/0 conn(0x7fd9d80a98f0 legacy=0x7fd9d8138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 >> v1:172.21.15.19:6789/0 conn(0x7fd9d81430c0 legacy=0x7fd9d81d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9d4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:36:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9d4ff9640 1 monclient: found mon.b 2024-03-25T19:36:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9d4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d81d83d0 con 0x7fd9d813f710 2024-03-25T19:36:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9d4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:36:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9d4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:36:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9de915640 5 monclient: authenticate success, global_id 4552 2024-03-25T19:36:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9de915640 10 monclient: _renew_subs 2024-03-25T19:36:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9de915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9de915640 1 -- 172.21.15.19:0/1184046333 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d81d41e0 con 0x7fd9d813f710 2024-03-25T19:36:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9de915640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9de915640 10 monclient: _renew_subs 2024-03-25T19:36:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9de915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9de915640 1 -- 172.21.15.19:0/1184046333 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d81d5110 con 0x7fd9d813f710 2024-03-25T19:36:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.878+0000 7fd9de915640 1 librados: init done 2024-03-25T19:36:27.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.879+0000 7fd9de915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:27.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.879+0000 7fd9de915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:27.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.879+0000 7fd9de915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:27.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:27.879+0000 7fd9de915640 1 -- 172.21.15.19:0/1184046333 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d814f320 con 0x7fd9d813f710 2024-03-25T19:36:28.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.169+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9cc0011f0 con 0x7fd9d813f710 2024-03-25T19:36:28.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.169+0000 7fd9d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:28.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.169+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fd9cc004c30 con 0x7fd9d813f710 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.169+0000 7fd9d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.169+0000 7fd9d4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.169+0000 7fd9d4ff9640 10 monclient: dump: 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:28.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.169+0000 7fd9d67fc640 4 set_mon_vals no callback set 2024-03-25T19:36:28.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.170+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fd9cc01ac40 con 0x7fd9d813f710 2024-03-25T19:36:28.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.170+0000 7fd9d7fff640 10 monclient: get_auth_request con 0x7fd9c406ded0 auth_method 0 2024-03-25T19:36:28.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.171+0000 7fd9d4ff9640 10 client.4552.objecter ms_handle_connect 0x7fd9c406ded0 2024-03-25T19:36:28.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.177+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fd9cc0548f0 con 0x7fd9d813f710 2024-03-25T19:36:28.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.177+0000 7fd9d4ff9640 10 client.4552.objecter ms_dispatch 0x7fd9d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:28.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.177+0000 7fd9d4ff9640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:28.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.177+0000 7fd9d4ff9640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:28.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.177+0000 7fd9d4ff9640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-25T19:36:28.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.178+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9cc050230 con 0x7fd9d813f710 2024-03-25T19:36:28.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.178+0000 7fd9d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:28.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.178+0000 7fd9d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:28.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.436+0000 7fd9de915640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:28.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.436+0000 7fd9de915640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:28.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.436+0000 7fd9de915640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:28.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.436+0000 7fd9de915640 1 -- 172.21.15.19:0/1184046333 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9d81d7930 con 0x7fd9d813f710 2024-03-25T19:36:28.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.484+0000 7fd9d4ff9640 1 -- 172.21.15.19:0/1184046333 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fd9cc052330 con 0x7fd9d813f710 2024-03-25T19:36:28.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.485+0000 7fd9d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:28.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.485+0000 7fd9d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:28.485 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:36:28.485 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:28.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.486+0000 7fd9de915640 10 librados: watch_flush enter 2024-03-25T19:36:28.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.486+0000 7fd9de915640 10 librados: watch_flush exit 2024-03-25T19:36:28.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.486+0000 7fd9de915640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-25T19:36:28.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.486+0000 7fd9de915640 10 client.4552.objecter successfully canceled tick 2024-03-25T19:36:28.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.487+0000 7fd9de915640 1 -- 172.21.15.19:0/1184046333 >> v1:172.21.15.188:6800/2207506229 conn(0x7fd9c406ded0 legacy=0x7fd9c4070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:28.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.487+0000 7fd9de915640 10 monclient: shutdown 2024-03-25T19:36:28.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.487+0000 7fd9de915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:28.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.487+0000 7fd9de915640 1 -- 172.21.15.19:0/1184046333 >> v1:172.21.15.188:6789/0 conn(0x7fd9d813f710 legacy=0x7fd9d81d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:28.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.487+0000 7fd9de915640 1 -- 172.21.15.19:0/1184046333 shutdown_connections 2024-03-25T19:36:28.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.487+0000 7fd9de915640 1 -- 172.21.15.19:0/1184046333 >> 172.21.15.19:0/1184046333 conn(0x7fd9d80b7ea0 msgr2=0x7fd9d813ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:28.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.487+0000 7fd9de915640 1 -- 172.21.15.19:0/1184046333 shutdown_connections 2024-03-25T19:36:28.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.487+0000 7fd9de915640 1 -- 172.21.15.19:0/1184046333 wait complete. 2024-03-25T19:36:28.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:28.487+0000 7fd9de915640 1 librados: shutdown 2024-03-25T19:36:28.497 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:35.304 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:35.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.421+0000 7fa5bf3f6640 10 monclient: get_monmap_and_config 2024-03-25T19:36:35.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.421+0000 7fa5bf3f6640 10 monclient: build_initial_monmap 2024-03-25T19:36:35.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.421+0000 7fa5bf3f6640 10 monclient: monmap: 2024-03-25T19:36:35.421 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:35.421 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:35.421 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:36:35.422502+0000 2024-03-25T19:36:35.421 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:36:35.422502+0000 2024-03-25T19:36:35.421 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:35.422 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:35.422 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:35.422 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:36:35.422 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:36:35.422 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:35.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.421+0000 7fa5bf3f6640 1 Processor -- start 2024-03-25T19:36:35.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.422+0000 7fa5bf3f6640 1 -- start start 2024-03-25T19:36:35.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.422+0000 7fa5bf3f6640 10 monclient: init 2024-03-25T19:36:35.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.422+0000 7fa5bf3f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:35.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.422+0000 7fa5bf3f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:35.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.422+0000 7fa5bf3f6640 10 monclient(hunting): picked mon.noname-b con 0x7fa5b80b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:35.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.422+0000 7fa5bf3f6640 10 monclient(hunting): picked mon.noname-a con 0x7fa5b80b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:35.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.422+0000 7fa5bf3f6640 10 monclient(hunting): picked mon.noname-c con 0x7fa5b814be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:35.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.422+0000 7fa5bf3f6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b80af940 con 0x7fa5b80b1730 2024-03-25T19:36:35.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.422+0000 7fa5bf3f6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b80b0a70 con 0x7fa5b814be90 2024-03-25T19:36:35.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.422+0000 7fa5bf3f6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b80b0d50 con 0x7fa5b80b4d60 2024-03-25T19:36:35.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.422+0000 7fa5bf3f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:35.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.422+0000 7fa5bf3f6640 10 monclient(hunting): authenticate will time out at 6448.692700s 2024-03-25T19:36:35.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.426+0000 7fa5bd192640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fa5b80b4d60 0x7fa5b813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:35240/0 (socket says 172.21.15.19:35240) 2024-03-25T19:36:35.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.426+0000 7fa5bd192640 1 -- 172.21.15.19:0/365586160 learned_addr learned my addr 172.21.15.19:0/365586160 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:35.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.436+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386258562 0 0) 0x7fa5b80b0a70 con 0x7fa5b814be90 2024-03-25T19:36:35.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.436+0000 7fa5affff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.436+0000 7fa5affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.436+0000 7fa5affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.436+0000 7fa5affff640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:36:35.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.436+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a4003520 con 0x7fa5b814be90 2024-03-25T19:36:35.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.440+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695096254 0 0) 0x7fa5b80b0d50 con 0x7fa5b80b4d60 2024-03-25T19:36:35.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.440+0000 7fa5affff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.440+0000 7fa5affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.440+0000 7fa5affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.440+0000 7fa5affff640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:36:35.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.440+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b80b0a70 con 0x7fa5b80b4d60 2024-03-25T19:36:35.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.444+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769800285 0 0) 0x7fa5b80af940 con 0x7fa5b80b1730 2024-03-25T19:36:35.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.444+0000 7fa5affff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.444+0000 7fa5affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.444+0000 7fa5affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.444+0000 7fa5affff640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:36:35.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.444+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b80b0d50 con 0x7fa5b80b1730 2024-03-25T19:36:35.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2502432708 0 0) 0x7fa5a4003520 con 0x7fa5b814be90 2024-03-25T19:36:35.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 >> v1:172.21.15.19:6790/0 conn(0x7fa5b80b4d60 legacy=0x7fa5b813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 >> v1:172.21.15.19:6789/0 conn(0x7fa5b80b1730 legacy=0x7fa5b80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:35.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 1 monclient: found mon.noname-c 2024-03-25T19:36:35.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:35.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b80b1030 con 0x7fa5b814be90 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 10 monclient: _finish_auth 0 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa5b0001690 con 0x7fa5b814be90 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5affff640 10 monclient: dump: 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:35.452 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.453 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:35.453 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:35.453 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:35.453 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:35.453 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:35.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5bf3f6640 5 monclient: authenticate success, global_id 4555 2024-03-25T19:36:35.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.451+0000 7fa5bf3f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:35.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.462+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5b00017f0 con 0x7fa5b814be90 2024-03-25T19:36:35.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.462+0000 7fa5affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:35.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.462+0000 7fa5affff640 1 -- 172.21.15.19:0/365586160 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa5b0001c00 con 0x7fa5b814be90 2024-03-25T19:36:35.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.462+0000 7fa5bf3f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:35.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.462+0000 7fa5bf3f6640 4 set_mon_vals no callback set 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.462+0000 7fa5affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.462+0000 7fa5affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.462+0000 7fa5affff640 10 monclient: dump: 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:35.463 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:35.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.463+0000 7fa5bf3f6640 10 monclient: shutdown 2024-03-25T19:36:35.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.463+0000 7fa5bf3f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:35.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.463+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/365586160 >> v1:172.21.15.188:6789/0 conn(0x7fa5b814be90 legacy=0x7fa5b814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.463+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/365586160 shutdown_connections 2024-03-25T19:36:35.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.463+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/365586160 >> 172.21.15.19:0/365586160 conn(0x7fa5b80617a0 msgr2=0x7fa5b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:35.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.463+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/365586160 shutdown_connections 2024-03-25T19:36:35.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.463+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/365586160 wait complete. 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.464+0000 7fa5bf3f6640 10 monclient: build_initial_monmap 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.464+0000 7fa5bf3f6640 10 monclient: monmap: 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.464+0000 7fa5bf3f6640 1 librados: starting msgr at 2024-03-25T19:36:35.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.464+0000 7fa5bf3f6640 1 librados: starting objecter 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.465+0000 7fa5bf3f6640 1 Processor -- start 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.465+0000 7fa5bf3f6640 1 -- start start 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.465+0000 7fa5bf3f6640 1 librados: setting wanted keys 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.465+0000 7fa5bf3f6640 1 librados: calling monclient init 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.465+0000 7fa5bf3f6640 10 monclient: init 2024-03-25T19:36:35.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.466+0000 7fa5bf3f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.466+0000 7fa5bf3f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.466+0000 7fa5bf3f6640 10 monclient(hunting): picked mon.noname-c con 0x7fa5b80b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.466+0000 7fa5bf3f6640 10 monclient(hunting): picked mon.noname-b con 0x7fa5b80b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.466+0000 7fa5bf3f6640 10 monclient(hunting): picked mon.noname-a con 0x7fa5b814be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.466+0000 7fa5bf3f6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b8140910 con 0x7fa5b814be90 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.466+0000 7fa5bf3f6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b8135810 con 0x7fa5b80b4d60 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.466+0000 7fa5bf3f6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b81456c0 con 0x7fa5b80b1730 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.466+0000 7fa5bf3f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:35.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.466+0000 7fa5bf3f6640 10 monclient(hunting): authenticate will time out at 6448.736197s 2024-03-25T19:36:35.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.470+0000 7fa5bc991640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fa5b80b4d60 0x7fa5b81349f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:59346/0 (socket says 172.21.15.19:59346) 2024-03-25T19:36:35.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.470+0000 7fa5bc991640 1 -- 172.21.15.19:0/650085958 learned_addr learned my addr 172.21.15.19:0/650085958 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:35.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.472+0000 7fa5adffb640 10 client.?.objecter ms_handle_connect 0x7fa5b80b4d60 2024-03-25T19:36:35.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.472+0000 7fa5adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:35.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.479+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251057938 0 0) 0x7fa5b8135810 con 0x7fa5b80b4d60 2024-03-25T19:36:35.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.479+0000 7fa5adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:35.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.479+0000 7fa5adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:35.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.479+0000 7fa5adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:35.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.479+0000 7fa5adffb640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:36:35.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.479+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa59c004eb0 con 0x7fa5b80b4d60 2024-03-25T19:36:35.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.487+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2408013495 0 0) 0x7fa59c004eb0 con 0x7fa5b80b4d60 2024-03-25T19:36:35.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.487+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5b8135810 con 0x7fa5b80b4d60 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.487+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa5a8001a10 con 0x7fa5b80b4d60 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.487+0000 7fa5adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.487+0000 7fa5adffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.487+0000 7fa5adffb640 10 monclient(hunting): dump: 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:35.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.487+0000 7fa5adffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:35.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.489+0000 7fa5adffb640 10 client.?.objecter ms_handle_connect 0x7fa5b80b1730 2024-03-25T19:36:35.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.489+0000 7fa5adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:35.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.490+0000 7fa5adffb640 10 client.?.objecter ms_handle_connect 0x7fa5b814be90 2024-03-25T19:36:35.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.490+0000 7fa5adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:35.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 299606322 0 0) 0x7fa5b8135810 con 0x7fa5b80b4d60 2024-03-25T19:36:35.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 >> v1:172.21.15.19:6790/0 conn(0x7fa5b80b1730 legacy=0x7fa5b81d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 >> v1:172.21.15.19:6789/0 conn(0x7fa5b814be90 legacy=0x7fa5b8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5adffb640 10 monclient: _finish_hunting 0 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5adffb640 1 monclient: found mon.b 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5adffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b8145850 con 0x7fa5b80b4d60 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5adffb640 10 monclient: _finish_auth 0 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5adffb640 10 monclient: _check_auth_tickets 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5bf3f6640 5 monclient: authenticate success, global_id 4558 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5bf3f6640 10 monclient: _renew_subs 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5bf3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/650085958 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b8140440 con 0x7fa5b80b4d60 2024-03-25T19:36:35.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5bf3f6640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:35.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5bf3f6640 10 monclient: _renew_subs 2024-03-25T19:36:35.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5bf3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:35.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/650085958 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b8144eb0 con 0x7fa5b80b4d60 2024-03-25T19:36:35.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.493+0000 7fa5bf3f6640 1 librados: init done 2024-03-25T19:36:35.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.494+0000 7fa5bf3f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:35.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.494+0000 7fa5bf3f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:35.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.494+0000 7fa5bf3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:35.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.494+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/650085958 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b80627b0 con 0x7fa5b80b4d60 2024-03-25T19:36:35.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.776+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5a8001d30 con 0x7fa5b80b4d60 2024-03-25T19:36:35.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.776+0000 7fa5adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:35.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.776+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa5a80038c0 con 0x7fa5b80b4d60 2024-03-25T19:36:35.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.776+0000 7fa5adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.776+0000 7fa5adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:35.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.776+0000 7fa5adffb640 10 monclient: dump: 2024-03-25T19:36:35.777 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:35.777 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:35.777 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:35.777 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:35.777 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.777 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:35.777 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:35.778 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:35.778 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:35.778 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:35.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.776+0000 7fa5af7fe640 4 set_mon_vals no callback set 2024-03-25T19:36:35.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.777+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fa5a8003ae0 con 0x7fa5b80b4d60 2024-03-25T19:36:35.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.778+0000 7fa5bd192640 10 monclient: get_auth_request con 0x7fa59c06dcc0 auth_method 0 2024-03-25T19:36:35.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.778+0000 7fa5adffb640 10 client.4558.objecter ms_handle_connect 0x7fa59c06dcc0 2024-03-25T19:36:35.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.782+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fa5a8054760 con 0x7fa5b80b4d60 2024-03-25T19:36:35.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.782+0000 7fa5adffb640 10 client.4558.objecter ms_dispatch 0x7fa5b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:35.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.782+0000 7fa5adffb640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:35.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.782+0000 7fa5adffb640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:35.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.782+0000 7fa5adffb640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-25T19:36:35.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.782+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5a8050270 con 0x7fa5b80b4d60 2024-03-25T19:36:35.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.782+0000 7fa5adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:35.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:35.782+0000 7fa5adffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:36.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.025+0000 7fa5bf3f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:36.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.025+0000 7fa5bf3f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:36.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.025+0000 7fa5bf3f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:36.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.025+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/650085958 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5b8069f40 con 0x7fa5b80b4d60 2024-03-25T19:36:36.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.048+0000 7fa5adffb640 1 -- 172.21.15.19:0/650085958 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fa5a80521a0 con 0x7fa5b80b4d60 2024-03-25T19:36:36.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.048+0000 7fa5adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:36.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.048+0000 7fa5adffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:36.048 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:36:36.049 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:36.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.050+0000 7fa5bf3f6640 10 librados: watch_flush enter 2024-03-25T19:36:36.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.050+0000 7fa5bf3f6640 10 librados: watch_flush exit 2024-03-25T19:36:36.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.050+0000 7fa5bf3f6640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-25T19:36:36.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.050+0000 7fa5bf3f6640 10 client.4558.objecter successfully canceled tick 2024-03-25T19:36:36.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.050+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/650085958 >> v1:172.21.15.188:6800/2207506229 conn(0x7fa59c06dcc0 legacy=0x7fa59c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.051+0000 7fa5bf3f6640 10 monclient: shutdown 2024-03-25T19:36:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.051+0000 7fa5bf3f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.051+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/650085958 >> v1:172.21.15.188:6789/0 conn(0x7fa5b80b4d60 legacy=0x7fa5b81349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.051+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/650085958 shutdown_connections 2024-03-25T19:36:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.051+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/650085958 >> 172.21.15.19:0/650085958 conn(0x7fa5b80617a0 msgr2=0x7fa5b80b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.051+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/650085958 shutdown_connections 2024-03-25T19:36:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.051+0000 7fa5bf3f6640 1 -- 172.21.15.19:0/650085958 wait complete. 2024-03-25T19:36:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:36.051+0000 7fa5bf3f6640 1 librados: shutdown 2024-03-25T19:36:36.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:38.080 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:00.809 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:38.092 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:00.821 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:38.129 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:00.858 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:38.138 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:08:00.840 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:38.145 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:08:00.848 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:38.151 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:00.881 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:38.157 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:00.886 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:38.174 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:08:00.877 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:38.190 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:00.919 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:42.869 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:42.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.968+0000 7f0a5999a640 10 monclient: get_monmap_and_config 2024-03-25T19:36:42.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.968+0000 7f0a5999a640 10 monclient: build_initial_monmap 2024-03-25T19:36:42.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.968+0000 7f0a5999a640 10 monclient: monmap: 2024-03-25T19:36:42.969 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:42.969 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:42.969 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:36:42.969890+0000 2024-03-25T19:36:42.969 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:36:42.969890+0000 2024-03-25T19:36:42.969 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:42.969 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:42.969 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:42.970 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:36:42.970 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:36:42.970 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:42.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 1 Processor -- start 2024-03-25T19:36:42.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 1 -- start start 2024-03-25T19:36:42.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 10 monclient: init 2024-03-25T19:36:42.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:42.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:42.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 10 monclient(hunting): picked mon.noname-c con 0x7f0a540b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:42.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 10 monclient(hunting): picked mon.noname-b con 0x7f0a540b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:42.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 10 monclient(hunting): picked mon.noname-a con 0x7f0a5414be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:42.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a540af940 con 0x7f0a5414be90 2024-03-25T19:36:42.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a540b0a70 con 0x7f0a540b4d60 2024-03-25T19:36:42.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a540b0d50 con 0x7f0a540b1730 2024-03-25T19:36:42.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:42.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.969+0000 7f0a5999a640 10 monclient(hunting): authenticate will time out at 6456.239870s 2024-03-25T19:36:42.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.978+0000 7f0a52ffd640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f0a540b4d60 0x7f0a5413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:54056/0 (socket says 172.21.15.19:54056) 2024-03-25T19:36:42.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.978+0000 7f0a52ffd640 1 -- 172.21.15.19:0/1119649178 learned_addr learned my addr 172.21.15.19:0/1119649178 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:42.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.992+0000 7f0a527fc640 1 -- 172.21.15.19:0/1119649178 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753170943 0 0) 0x7f0a540b0a70 con 0x7f0a540b4d60 2024-03-25T19:36:42.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.992+0000 7f0a527fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.992+0000 7f0a527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.992+0000 7f0a527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.992+0000 7f0a527fc640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:36:42.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:42.992+0000 7f0a527fc640 1 -- 172.21.15.19:0/1119649178 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a40003520 con 0x7f0a540b4d60 2024-03-25T19:36:43.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 1 -- 172.21.15.19:0/1119649178 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1176165165 0 0) 0x7f0a40003520 con 0x7f0a540b4d60 2024-03-25T19:36:43.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 1 -- 172.21.15.19:0/1119649178 >> v1:172.21.15.19:6790/0 conn(0x7f0a540b1730 legacy=0x7f0a540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:36:43.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 1 -- 172.21.15.19:0/1119649178 >> v1:172.21.15.19:6789/0 conn(0x7f0a5414be90 legacy=0x7f0a5414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:43.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 1 monclient: found mon.noname-c 2024-03-25T19:36:43.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:43.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 1 -- 172.21.15.19:0/1119649178 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a540b1030 con 0x7f0a540b4d60 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 1 -- 172.21.15.19:0/1119649178 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0a44001690 con 0x7f0a540b4d60 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a5999a640 5 monclient: authenticate success, global_id 4561 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a5999a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a527fc640 10 monclient: dump: 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:43.009 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:43.010 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:43.010 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:43.010 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:43.010 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:43.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.008+0000 7f0a5999a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.012+0000 7f0a527fc640 1 -- 172.21.15.19:0/1119649178 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a44001860 con 0x7f0a540b4d60 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.012+0000 7f0a527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.012+0000 7f0a527fc640 1 -- 172.21.15.19:0/1119649178 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0a44001cb0 con 0x7f0a540b4d60 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.012+0000 7f0a5999a640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.012+0000 7f0a5999a640 4 set_mon_vals no callback set 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.012+0000 7f0a527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.012+0000 7f0a527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.012+0000 7f0a527fc640 10 monclient: dump: 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:43.013 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:43.014 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:43.014 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:43.014 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:43.014 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:43.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.013+0000 7f0a5999a640 10 monclient: shutdown 2024-03-25T19:36:43.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.013+0000 7f0a5999a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:43.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.013+0000 7f0a5999a640 1 -- 172.21.15.19:0/1119649178 >> v1:172.21.15.188:6789/0 conn(0x7f0a540b4d60 legacy=0x7f0a5413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.013+0000 7f0a5999a640 1 -- 172.21.15.19:0/1119649178 shutdown_connections 2024-03-25T19:36:43.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.013+0000 7f0a5999a640 1 -- 172.21.15.19:0/1119649178 >> 172.21.15.19:0/1119649178 conn(0x7f0a540617a0 msgr2=0x7f0a54061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:43.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.013+0000 7f0a5999a640 1 -- 172.21.15.19:0/1119649178 shutdown_connections 2024-03-25T19:36:43.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.013+0000 7f0a5999a640 1 -- 172.21.15.19:0/1119649178 wait complete. 2024-03-25T19:36:43.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.014+0000 7f0a5999a640 10 monclient: build_initial_monmap 2024-03-25T19:36:43.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.014+0000 7f0a5999a640 10 monclient: monmap: 2024-03-25T19:36:43.016 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:43.016 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.014+0000 7f0a5999a640 1 librados: starting msgr at 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.014+0000 7f0a5999a640 1 librados: starting objecter 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.014+0000 7f0a5999a640 1 Processor -- start 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 1 -- start start 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 1 librados: setting wanted keys 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 1 librados: calling monclient init 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 10 monclient: init 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:43.017 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:43.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 10 monclient(hunting): picked mon.noname-c con 0x7f0a540b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:43.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 10 monclient(hunting): picked mon.noname-b con 0x7f0a540b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:43.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 10 monclient(hunting): picked mon.noname-a con 0x7f0a5414be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:43.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a541407f0 con 0x7f0a5414be90 2024-03-25T19:36:43.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a5413b080 con 0x7f0a540b4d60 2024-03-25T19:36:43.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a54140380 con 0x7f0a540b1730 2024-03-25T19:36:43.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:43.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.015+0000 7f0a5999a640 10 monclient(hunting): authenticate will time out at 6456.285694s 2024-03-25T19:36:43.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.026+0000 7f0a4a7fc640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f0a540b4d60 0x7f0a541dab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:54062/0 (socket says 172.21.15.19:54062) 2024-03-25T19:36:43.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.026+0000 7f0a4a7fc640 1 -- 172.21.15.19:0/713052375 learned_addr learned my addr 172.21.15.19:0/713052375 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:43.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.029+0000 7f0a50ff9640 10 client.?.objecter ms_handle_connect 0x7f0a540b4d60 2024-03-25T19:36:43.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.029+0000 7f0a50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:43.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.030+0000 7f0a50ff9640 10 client.?.objecter ms_handle_connect 0x7f0a540b1730 2024-03-25T19:36:43.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.030+0000 7f0a50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:43.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.043+0000 7f0a50ff9640 10 client.?.objecter ms_handle_connect 0x7f0a5414be90 2024-03-25T19:36:43.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.043+0000 7f0a50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:43.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.044+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137888928 0 0) 0x7f0a5413b080 con 0x7f0a540b4d60 2024-03-25T19:36:43.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.044+0000 7f0a50ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:43.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.044+0000 7f0a50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:43.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.044+0000 7f0a50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:43.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.044+0000 7f0a50ff9640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:36:43.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.044+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a34004eb0 con 0x7f0a540b4d60 2024-03-25T19:36:43.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.047+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407060949 0 0) 0x7f0a54140380 con 0x7f0a540b1730 2024-03-25T19:36:43.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.047+0000 7f0a50ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:43.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.047+0000 7f0a50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:43.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.047+0000 7f0a50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:43.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.047+0000 7f0a50ff9640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:36:43.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.047+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a5413b080 con 0x7f0a540b1730 2024-03-25T19:36:43.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.055+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2785157751 0 0) 0x7f0a34004eb0 con 0x7f0a540b4d60 2024-03-25T19:36:43.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.055+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a54140380 con 0x7f0a540b4d60 2024-03-25T19:36:43.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.055+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0a3c001960 con 0x7f0a540b4d60 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.055+0000 7f0a50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.055+0000 7f0a50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.055+0000 7f0a50ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:43.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.055+0000 7f0a50ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:43.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.070+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822101839 0 0) 0x7f0a541407f0 con 0x7f0a5414be90 2024-03-25T19:36:43.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.070+0000 7f0a50ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:43.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.070+0000 7f0a50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:43.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.070+0000 7f0a50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:43.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.070+0000 7f0a50ff9640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:36:43.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.070+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a34004eb0 con 0x7f0a5414be90 2024-03-25T19:36:43.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.076+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 731464552 0 0) 0x7f0a5413b080 con 0x7f0a540b1730 2024-03-25T19:36:43.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.076+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a541407f0 con 0x7f0a540b1730 2024-03-25T19:36:43.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.076+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0a44001b70 con 0x7f0a540b1730 2024-03-25T19:36:43.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.076+0000 7f0a50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:43.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.076+0000 7f0a50ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:43.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.076+0000 7f0a50ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:43.076 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:43.076 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:43.076 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:43.077 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:43.077 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:43.077 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:43.077 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:43.077 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:43.077 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:43.077 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:43.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2900967155 0 0) 0x7f0a54140380 con 0x7f0a540b4d60 2024-03-25T19:36:43.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 >> v1:172.21.15.19:6790/0 conn(0x7f0a540b1730 legacy=0x7f0a54134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 >> v1:172.21.15.19:6789/0 conn(0x7f0a5414be90 legacy=0x7f0a541db230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a50ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a50ff9640 1 monclient: found mon.b 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a50ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a54140510 con 0x7f0a540b4d60 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a50ff9640 10 monclient: _finish_auth 0 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a50ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a5999a640 5 monclient: authenticate success, global_id 4564 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a5999a640 10 monclient: _renew_subs 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a5999a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a5999a640 1 -- 172.21.15.19:0/713052375 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a541355c0 con 0x7f0a540b4d60 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a5999a640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a5999a640 10 monclient: _renew_subs 2024-03-25T19:36:43.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a5999a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:43.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a5999a640 1 -- 172.21.15.19:0/713052375 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a54145600 con 0x7f0a540b4d60 2024-03-25T19:36:43.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.083+0000 7f0a5999a640 1 librados: init done 2024-03-25T19:36:43.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.084+0000 7f0a5999a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:43.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.084+0000 7f0a5999a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:43.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.084+0000 7f0a5999a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:43.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.084+0000 7f0a5999a640 1 -- 172.21.15.19:0/713052375 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a540627b0 con 0x7f0a540b4d60 2024-03-25T19:36:43.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.371+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a3c0035f0 con 0x7f0a540b4d60 2024-03-25T19:36:43.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.371+0000 7f0a50ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:43.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.371+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0a3c003940 con 0x7f0a540b4d60 2024-03-25T19:36:43.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.371+0000 7f0a50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:43.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.371+0000 7f0a50ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:43.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.371+0000 7f0a50ff9640 10 monclient: dump: 2024-03-25T19:36:43.371 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:43.371 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:43.371 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:43.371 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:43.371 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:43.372 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:43.372 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:43.372 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:43.372 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:43.372 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:43.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.371+0000 7f0a51ffb640 4 set_mon_vals no callback set 2024-03-25T19:36:43.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.371+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f0a3c003b60 con 0x7f0a540b4d60 2024-03-25T19:36:43.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.372+0000 7f0a52ffd640 10 monclient: get_auth_request con 0x7f0a3406dfb0 auth_method 0 2024-03-25T19:36:43.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.372+0000 7f0a50ff9640 10 client.4564.objecter ms_handle_connect 0x7f0a3406dfb0 2024-03-25T19:36:43.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.377+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f0a3c054780 con 0x7f0a540b4d60 2024-03-25T19:36:43.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.377+0000 7f0a50ff9640 10 client.4564.objecter ms_dispatch 0x7f0a54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:43.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.377+0000 7f0a50ff9640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:43.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.377+0000 7f0a50ff9640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:43.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.377+0000 7f0a50ff9640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T19:36:43.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.378+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0a3c050290 con 0x7f0a540b4d60 2024-03-25T19:36:43.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.378+0000 7f0a50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:43.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.378+0000 7f0a50ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:43.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.639+0000 7f0a5999a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:43.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.639+0000 7f0a5999a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:43.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.639+0000 7f0a5999a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:43.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.639+0000 7f0a5999a640 1 -- 172.21.15.19:0/713052375 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a54135750 con 0x7f0a540b4d60 2024-03-25T19:36:43.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.683+0000 7f0a50ff9640 1 -- 172.21.15.19:0/713052375 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f0a3c0521c0 con 0x7f0a540b4d60 2024-03-25T19:36:43.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.683+0000 7f0a50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:43.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.683+0000 7f0a50ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:43.683 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:36:43.684 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:43.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.685+0000 7f0a5999a640 10 librados: watch_flush enter 2024-03-25T19:36:43.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.685+0000 7f0a5999a640 10 librados: watch_flush exit 2024-03-25T19:36:43.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.685+0000 7f0a5999a640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T19:36:43.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.685+0000 7f0a5999a640 10 client.4564.objecter successfully canceled tick 2024-03-25T19:36:43.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.685+0000 7f0a5999a640 1 -- 172.21.15.19:0/713052375 >> v1:172.21.15.188:6800/2207506229 conn(0x7f0a3406dfb0 legacy=0x7f0a34070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.685+0000 7f0a5999a640 10 monclient: shutdown 2024-03-25T19:36:43.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.685+0000 7f0a5999a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:43.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.685+0000 7f0a5999a640 1 -- 172.21.15.19:0/713052375 >> v1:172.21.15.188:6789/0 conn(0x7f0a540b4d60 legacy=0x7f0a541dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.685+0000 7f0a5999a640 1 -- 172.21.15.19:0/713052375 shutdown_connections 2024-03-25T19:36:43.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.686+0000 7f0a5999a640 1 -- 172.21.15.19:0/713052375 >> 172.21.15.19:0/713052375 conn(0x7f0a540617a0 msgr2=0x7f0a5413c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:43.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.686+0000 7f0a5999a640 1 -- 172.21.15.19:0/713052375 shutdown_connections 2024-03-25T19:36:43.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.686+0000 7f0a5999a640 1 -- 172.21.15.19:0/713052375 wait complete. 2024-03-25T19:36:43.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:43.686+0000 7f0a5999a640 1 librados: shutdown 2024-03-25T19:36:43.699 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:50.506 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:50.617 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.618+0000 7efc83dd4640 10 monclient: get_monmap_and_config 2024-03-25T19:36:50.617 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.618+0000 7efc83dd4640 10 monclient: build_initial_monmap 2024-03-25T19:36:50.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.618+0000 7efc83dd4640 10 monclient: monmap: 2024-03-25T19:36:50.618 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:50.618 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:50.618 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:36:50.619056+0000 2024-03-25T19:36:50.618 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:36:50.619056+0000 2024-03-25T19:36:50.618 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:50.618 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:50.618 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:50.618 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:36:50.618 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:36:50.618 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:50.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.619+0000 7efc83dd4640 1 Processor -- start 2024-03-25T19:36:50.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.619+0000 7efc83dd4640 1 -- start start 2024-03-25T19:36:50.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.619+0000 7efc83dd4640 10 monclient: init 2024-03-25T19:36:50.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.620+0000 7efc83dd4640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:50.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.620+0000 7efc83dd4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:50.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.620+0000 7efc83dd4640 10 monclient(hunting): picked mon.noname-a con 0x7efc7c0b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:50.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.620+0000 7efc83dd4640 10 monclient(hunting): picked mon.noname-b con 0x7efc7c0b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:50.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.620+0000 7efc83dd4640 10 monclient(hunting): picked mon.noname-c con 0x7efc7c14be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:50.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.620+0000 7efc83dd4640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c0af940 con 0x7efc7c0b4d60 2024-03-25T19:36:50.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.620+0000 7efc83dd4640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c0b0a70 con 0x7efc7c14be90 2024-03-25T19:36:50.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.620+0000 7efc83dd4640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c0b0d50 con 0x7efc7c0b1730 2024-03-25T19:36:50.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.620+0000 7efc83dd4640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:50.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.620+0000 7efc83dd4640 10 monclient(hunting): authenticate will time out at 6463.889574s 2024-03-25T19:36:50.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.624+0000 7efc8136f640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7efc7c0b1730 0x7efc7c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:47062/0 (socket says 172.21.15.19:47062) 2024-03-25T19:36:50.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.624+0000 7efc8136f640 1 -- 172.21.15.19:0/3585088227 learned_addr learned my addr 172.21.15.19:0/3585088227 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:50.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.631+0000 7efc80b6e640 1 -- 172.21.15.19:0/3585088227 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493321743 0 0) 0x7efc7c0b0d50 con 0x7efc7c0b1730 2024-03-25T19:36:50.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.631+0000 7efc80b6e640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.631+0000 7efc80b6e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.631+0000 7efc80b6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.631+0000 7efc80b6e640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:36:50.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.631+0000 7efc80b6e640 1 -- 172.21.15.19:0/3585088227 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc68003520 con 0x7efc7c0b1730 2024-03-25T19:36:50.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.644+0000 7efc80b6e640 1 -- 172.21.15.19:0/3585088227 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040903327 0 0) 0x7efc7c0b0a70 con 0x7efc7c14be90 2024-03-25T19:36:50.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.644+0000 7efc80b6e640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.644+0000 7efc80b6e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.644+0000 7efc80b6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.644+0000 7efc80b6e640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:36:50.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.644+0000 7efc80b6e640 1 -- 172.21.15.19:0/3585088227 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7c0b0d50 con 0x7efc7c14be90 2024-03-25T19:36:50.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 1 -- 172.21.15.19:0/3585088227 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2751130144 0 0) 0x7efc7c0b0d50 con 0x7efc7c14be90 2024-03-25T19:36:50.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 1 -- 172.21.15.19:0/3585088227 >> v1:172.21.15.19:6790/0 conn(0x7efc7c0b1730 legacy=0x7efc7c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 1 -- 172.21.15.19:0/3585088227 >> v1:172.21.15.19:6789/0 conn(0x7efc7c0b4d60 legacy=0x7efc7c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 10 monclient: _finish_hunting 0 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 1 monclient: found mon.noname-c 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 1 -- 172.21.15.19:0/3585088227 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c0b1030 con 0x7efc7c14be90 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 10 monclient: _finish_auth 0 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 10 monclient: _check_auth_tickets 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 1 -- 172.21.15.19:0/3585088227 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efc740019f0 con 0x7efc7c14be90 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc83dd4640 5 monclient: authenticate success, global_id 4567 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc83dd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc80b6e640 10 monclient: dump: 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:50.654 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:50.655 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:50.655 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.655 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:50.655 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:50.655 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:50.655 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:50.655 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:50.655 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.654+0000 7efc83dd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:50.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.680+0000 7efc80b6e640 1 -- 172.21.15.19:0/3585088227 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc74001b50 con 0x7efc7c14be90 2024-03-25T19:36:50.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.680+0000 7efc80b6e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:50.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.680+0000 7efc80b6e640 1 -- 172.21.15.19:0/3585088227 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efc74001fa0 con 0x7efc7c14be90 2024-03-25T19:36:50.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.680+0000 7efc80b6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.680+0000 7efc80b6e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:50.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.680+0000 7efc80b6e640 10 monclient: dump: 2024-03-25T19:36:50.679 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:50.679 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:50.679 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:50.679 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:50.679 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.679 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:50.680 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:50.680 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:50.680 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:50.680 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:50.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.680+0000 7efc83dd4640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:50.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.680+0000 7efc83dd4640 4 set_mon_vals no callback set 2024-03-25T19:36:50.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.680+0000 7efc83dd4640 10 monclient: shutdown 2024-03-25T19:36:50.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.680+0000 7efc83dd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:50.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.680+0000 7efc83dd4640 1 -- 172.21.15.19:0/3585088227 >> v1:172.21.15.188:6789/0 conn(0x7efc7c14be90 legacy=0x7efc7c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.681+0000 7efc83dd4640 1 -- 172.21.15.19:0/3585088227 shutdown_connections 2024-03-25T19:36:50.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.681+0000 7efc83dd4640 1 -- 172.21.15.19:0/3585088227 >> 172.21.15.19:0/3585088227 conn(0x7efc7c0617a0 msgr2=0x7efc7c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:50.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.681+0000 7efc83dd4640 1 -- 172.21.15.19:0/3585088227 shutdown_connections 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.681+0000 7efc83dd4640 1 -- 172.21.15.19:0/3585088227 wait complete. 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.682+0000 7efc83dd4640 10 monclient: build_initial_monmap 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.682+0000 7efc83dd4640 10 monclient: monmap: 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.682+0000 7efc83dd4640 1 librados: starting msgr at 2024-03-25T19:36:50.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.682+0000 7efc83dd4640 1 librados: starting objecter 2024-03-25T19:36:50.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.682+0000 7efc83dd4640 1 Processor -- start 2024-03-25T19:36:50.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.682+0000 7efc83dd4640 1 -- start start 2024-03-25T19:36:50.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.682+0000 7efc83dd4640 1 librados: setting wanted keys 2024-03-25T19:36:50.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.682+0000 7efc83dd4640 1 librados: calling monclient init 2024-03-25T19:36:50.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.682+0000 7efc83dd4640 10 monclient: init 2024-03-25T19:36:50.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.683+0000 7efc83dd4640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:50.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.683+0000 7efc83dd4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:50.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.683+0000 7efc83dd4640 10 monclient(hunting): picked mon.noname-c con 0x7efc7c0b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:50.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.683+0000 7efc83dd4640 10 monclient(hunting): picked mon.noname-a con 0x7efc7c0b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:50.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.683+0000 7efc83dd4640 10 monclient(hunting): picked mon.noname-b con 0x7efc7c14be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:50.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.683+0000 7efc83dd4640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c1d7980 con 0x7efc7c0b4d60 2024-03-25T19:36:50.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.683+0000 7efc83dd4640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c1d7c60 con 0x7efc7c14be90 2024-03-25T19:36:50.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.683+0000 7efc83dd4640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c0aa860 con 0x7efc7c0b1730 2024-03-25T19:36:50.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.683+0000 7efc83dd4640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:50.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.683+0000 7efc83dd4640 10 monclient(hunting): authenticate will time out at 6463.952520s 2024-03-25T19:36:50.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.686+0000 7efc82371640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7efc7c14be90 0x7efc7c138130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:50326/0 (socket says 172.21.15.19:50326) 2024-03-25T19:36:50.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.686+0000 7efc82371640 1 -- 172.21.15.19:0/3276792884 learned_addr learned my addr 172.21.15.19:0/3276792884 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:50.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.688+0000 7efc727fc640 10 client.?.objecter ms_handle_connect 0x7efc7c14be90 2024-03-25T19:36:50.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.688+0000 7efc727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:50.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.692+0000 7efc727fc640 10 client.?.objecter ms_handle_connect 0x7efc7c0b1730 2024-03-25T19:36:50.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.692+0000 7efc727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:50.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.693+0000 7efc727fc640 10 client.?.objecter ms_handle_connect 0x7efc7c0b4d60 2024-03-25T19:36:50.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.693+0000 7efc727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:50.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.695+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094382970 0 0) 0x7efc7c1d7c60 con 0x7efc7c14be90 2024-03-25T19:36:50.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.695+0000 7efc727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.695+0000 7efc727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.695+0000 7efc727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.695+0000 7efc727fc640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:36:50.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.695+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc60004eb0 con 0x7efc7c14be90 2024-03-25T19:36:50.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.702+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170775677 0 0) 0x7efc7c0aa860 con 0x7efc7c0b1730 2024-03-25T19:36:50.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.702+0000 7efc727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.702+0000 7efc727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.702+0000 7efc727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.702+0000 7efc727fc640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:36:50.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.702+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7c1d7c60 con 0x7efc7c0b1730 2024-03-25T19:36:50.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.703+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197238672 0 0) 0x7efc7c1d7980 con 0x7efc7c0b4d60 2024-03-25T19:36:50.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.703+0000 7efc727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.703+0000 7efc727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.703+0000 7efc727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.703+0000 7efc727fc640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:36:50.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.703+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7c0aa860 con 0x7efc7c0b4d60 2024-03-25T19:36:50.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.708+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1011476735 0 0) 0x7efc60004eb0 con 0x7efc7c14be90 2024-03-25T19:36:50.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.708+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc7c1d7980 con 0x7efc7c14be90 2024-03-25T19:36:50.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.708+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efc74001f60 con 0x7efc7c14be90 2024-03-25T19:36:50.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.708+0000 7efc727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.708+0000 7efc727fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:50.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.708+0000 7efc727fc640 10 monclient(hunting): dump: 2024-03-25T19:36:50.707 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:50.708 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:50.708 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:50.708 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:50.708 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.708 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:50.708 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:50.708 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:50.708 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:50.708 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:50.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.708+0000 7efc727fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:50.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.714+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2008092210 0 0) 0x7efc7c0aa860 con 0x7efc7c0b4d60 2024-03-25T19:36:50.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.714+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc60004eb0 con 0x7efc7c0b4d60 2024-03-25T19:36:50.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.714+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efc6c0017b0 con 0x7efc7c0b4d60 2024-03-25T19:36:50.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.714+0000 7efc727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.714+0000 7efc727fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.714+0000 7efc727fc640 10 monclient(hunting): dump: 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.714+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3173339827 0 0) 0x7efc7c1d7980 con 0x7efc7c14be90 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 >> v1:172.21.15.19:6790/0 conn(0x7efc7c0b1730 legacy=0x7efc7c1d80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 >> v1:172.21.15.19:6789/0 conn(0x7efc7c0b4d60 legacy=0x7efc7c137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc727fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:50.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc727fc640 1 monclient: found mon.b 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc727fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c0aaad0 con 0x7efc7c14be90 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc727fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc727fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc83dd4640 5 monclient: authenticate success, global_id 4570 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc83dd4640 10 monclient: _renew_subs 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc83dd4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc83dd4640 1 -- 172.21.15.19:0/3276792884 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c1388a0 con 0x7efc7c14be90 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc83dd4640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc83dd4640 10 monclient: _renew_subs 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc83dd4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc83dd4640 1 -- 172.21.15.19:0/3276792884 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c0a98c0 con 0x7efc7c14be90 2024-03-25T19:36:50.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.715+0000 7efc83dd4640 1 librados: init done 2024-03-25T19:36:50.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.716+0000 7efc83dd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:50.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.716+0000 7efc83dd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:50.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.716+0000 7efc83dd4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:50.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.716+0000 7efc83dd4640 1 -- 172.21.15.19:0/3276792884 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7c0b22c0 con 0x7efc7c14be90 2024-03-25T19:36:50.929 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.929+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc74003c20 con 0x7efc7c14be90 2024-03-25T19:36:50.929 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.929+0000 7efc727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:50.929 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.929+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7efc74003ff0 con 0x7efc7c14be90 2024-03-25T19:36:50.929 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.929+0000 7efc727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.929 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.929+0000 7efc727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:50.929 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.929+0000 7efc727fc640 10 monclient: dump: 2024-03-25T19:36:50.929 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:50.929 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:50.929 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:50.929 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:50.930 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.930 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:50.930 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:50.930 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:50.930 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:50.930 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:50.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.929+0000 7efc73fff640 4 set_mon_vals no callback set 2024-03-25T19:36:50.980 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.982+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7efc74004150 con 0x7efc7c14be90 2024-03-25T19:36:50.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.982+0000 7efc81b70640 10 monclient: get_auth_request con 0x7efc6006e000 auth_method 0 2024-03-25T19:36:50.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.983+0000 7efc727fc640 10 client.4570.objecter ms_handle_connect 0x7efc6006e000 2024-03-25T19:36:50.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.986+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7efc74054b40 con 0x7efc7c14be90 2024-03-25T19:36:50.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.986+0000 7efc727fc640 10 client.4570.objecter ms_dispatch 0x7efc7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:50.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.986+0000 7efc727fc640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:50.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.986+0000 7efc727fc640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:50.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.986+0000 7efc727fc640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T19:36:50.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.988+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc74050650 con 0x7efc7c14be90 2024-03-25T19:36:50.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.988+0000 7efc727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:50.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:50.988+0000 7efc727fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:51.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.217+0000 7efc83dd4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:51.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.217+0000 7efc83dd4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:51.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.217+0000 7efc83dd4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:36:51.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.217+0000 7efc83dd4640 1 -- 172.21.15.19:0/3276792884 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc7c0b8af0 con 0x7efc7c14be90 2024-03-25T19:36:51.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.237+0000 7efc727fc640 1 -- 172.21.15.19:0/3276792884 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7efc74052580 con 0x7efc7c14be90 2024-03-25T19:36:51.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.237+0000 7efc727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:51.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.237+0000 7efc727fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:51.237 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:36:51.237 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:51.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 10 librados: watch_flush enter 2024-03-25T19:36:51.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 10 librados: watch_flush exit 2024-03-25T19:36:51.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T19:36:51.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 10 client.4570.objecter successfully canceled tick 2024-03-25T19:36:51.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 1 -- 172.21.15.19:0/3276792884 >> v1:172.21.15.188:6800/2207506229 conn(0x7efc6006e000 legacy=0x7efc600704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:51.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 10 monclient: shutdown 2024-03-25T19:36:51.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:51.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 1 -- 172.21.15.19:0/3276792884 >> v1:172.21.15.188:6789/0 conn(0x7efc7c14be90 legacy=0x7efc7c138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:51.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 1 -- 172.21.15.19:0/3276792884 shutdown_connections 2024-03-25T19:36:51.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 1 -- 172.21.15.19:0/3276792884 >> 172.21.15.19:0/3276792884 conn(0x7efc7c0617a0 msgr2=0x7efc7c1450c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:51.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 1 -- 172.21.15.19:0/3276792884 shutdown_connections 2024-03-25T19:36:51.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 1 -- 172.21.15.19:0/3276792884 wait complete. 2024-03-25T19:36:51.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:51.240+0000 7efc83dd4640 1 librados: shutdown 2024-03-25T19:36:51.252 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:58.059 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:58.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.163+0000 7ff08effe640 10 monclient: get_monmap_and_config 2024-03-25T19:36:58.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.163+0000 7ff08effe640 10 monclient: build_initial_monmap 2024-03-25T19:36:58.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.163+0000 7ff08effe640 10 monclient: monmap: 2024-03-25T19:36:58.162 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:58.162 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:58.162 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:36:58.163408+0000 2024-03-25T19:36:58.162 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:36:58.163408+0000 2024-03-25T19:36:58.162 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:58.162 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:58.162 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:58.162 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:36:58.163 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:36:58.163 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.163+0000 7ff08effe640 1 Processor -- start 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.163+0000 7ff08effe640 1 -- start start 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.163+0000 7ff08effe640 10 monclient: init 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.164+0000 7ff08effe640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.164+0000 7ff08effe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.164+0000 7ff08effe640 10 monclient(hunting): picked mon.noname-b con 0x7ff0880b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.164+0000 7ff08effe640 10 monclient(hunting): picked mon.noname-a con 0x7ff0880b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.164+0000 7ff08effe640 10 monclient(hunting): picked mon.noname-c con 0x7ff08814be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.164+0000 7ff08effe640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0880af940 con 0x7ff0880b1730 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.164+0000 7ff08effe640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0880b0a70 con 0x7ff08814be90 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.164+0000 7ff08effe640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0880b0d50 con 0x7ff0880b4d60 2024-03-25T19:36:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.164+0000 7ff08effe640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:58.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.164+0000 7ff08effe640 10 monclient(hunting): authenticate will time out at 6471.433524s 2024-03-25T19:36:58.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.165+0000 7ff08cd9a640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7ff0880b4d60 0x7ff08813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:51976/0 (socket says 172.21.15.19:51976) 2024-03-25T19:36:58.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.165+0000 7ff08cd9a640 1 -- 172.21.15.19:0/1475346774 learned_addr learned my addr 172.21.15.19:0/1475346774 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:58.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.230+0000 7ff0877fe640 1 -- 172.21.15.19:0/1475346774 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071754555 0 0) 0x7ff0880af940 con 0x7ff0880b1730 2024-03-25T19:36:58.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.230+0000 7ff0877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.230+0000 7ff0877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.230+0000 7ff0877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.230+0000 7ff0877fe640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:36:58.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.230+0000 7ff0877fe640 1 -- 172.21.15.19:0/1475346774 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff078003520 con 0x7ff0880b1730 2024-03-25T19:36:58.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 1 -- 172.21.15.19:0/1475346774 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1788426357 0 0) 0x7ff078003520 con 0x7ff0880b1730 2024-03-25T19:36:58.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 1 -- 172.21.15.19:0/1475346774 >> v1:172.21.15.19:6790/0 conn(0x7ff0880b4d60 legacy=0x7ff08813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 1 -- 172.21.15.19:0/1475346774 >> v1:172.21.15.188:6789/0 conn(0x7ff08814be90 legacy=0x7ff08814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:58.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 1 monclient: found mon.noname-a 2024-03-25T19:36:58.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:58.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:36:58.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 1 -- 172.21.15.19:0/1475346774 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0880b1030 con 0x7ff0880b1730 2024-03-25T19:36:58.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 1 -- 172.21.15.19:0/1475346774 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff074001690 con 0x7ff0880b1730 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff0877fe640 10 monclient: dump: 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:58.242 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:58.243 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:58.243 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff08effe640 5 monclient: authenticate success, global_id 4584 2024-03-25T19:36:58.243 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.242+0000 7ff08effe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:58.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.265+0000 7ff0877fe640 1 -- 172.21.15.19:0/1475346774 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0740017f0 con 0x7ff0880b1730 2024-03-25T19:36:58.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.265+0000 7ff0877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:58.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.265+0000 7ff0877fe640 1 -- 172.21.15.19:0/1475346774 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff074001c40 con 0x7ff0880b1730 2024-03-25T19:36:58.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.265+0000 7ff08effe640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:58.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.265+0000 7ff08effe640 4 set_mon_vals no callback set 2024-03-25T19:36:58.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.265+0000 7ff0877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.265+0000 7ff0877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:58.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.265+0000 7ff0877fe640 10 monclient: dump: 2024-03-25T19:36:58.264 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:58.264 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:58.264 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:58.264 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:58.265 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.265 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:58.265 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:58.265 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:58.265 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:58.265 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:58.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.266+0000 7ff08effe640 10 monclient: shutdown 2024-03-25T19:36:58.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.266+0000 7ff08effe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.266+0000 7ff08effe640 1 -- 172.21.15.19:0/1475346774 >> v1:172.21.15.19:6789/0 conn(0x7ff0880b1730 legacy=0x7ff0880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.266+0000 7ff08effe640 1 -- 172.21.15.19:0/1475346774 shutdown_connections 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.266+0000 7ff08effe640 1 -- 172.21.15.19:0/1475346774 >> 172.21.15.19:0/1475346774 conn(0x7ff0880617a0 msgr2=0x7ff088061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.266+0000 7ff08effe640 1 -- 172.21.15.19:0/1475346774 shutdown_connections 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.266+0000 7ff08effe640 1 -- 172.21.15.19:0/1475346774 wait complete. 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.267+0000 7ff08effe640 10 monclient: build_initial_monmap 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.267+0000 7ff08effe640 10 monclient: monmap: 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:36:58.267 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:36:58.268 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:58.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.267+0000 7ff08effe640 1 librados: starting msgr at 2024-03-25T19:36:58.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.267+0000 7ff08effe640 1 librados: starting objecter 2024-03-25T19:36:58.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.267+0000 7ff08effe640 1 Processor -- start 2024-03-25T19:36:58.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 1 -- start start 2024-03-25T19:36:58.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 1 librados: setting wanted keys 2024-03-25T19:36:58.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 1 librados: calling monclient init 2024-03-25T19:36:58.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 10 monclient: init 2024-03-25T19:36:58.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:58.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:58.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 10 monclient(hunting): picked mon.noname-b con 0x7ff0880b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:36:58.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 10 monclient(hunting): picked mon.noname-a con 0x7ff0880b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:36:58.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 10 monclient(hunting): picked mon.noname-c con 0x7ff08814be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:36:58.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff088137b10 con 0x7ff0880b4d60 2024-03-25T19:36:58.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff088137f30 con 0x7ff0880b1730 2024-03-25T19:36:58.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0881380c0 con 0x7ff08814be90 2024-03-25T19:36:58.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:58.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.268+0000 7ff08effe640 10 monclient(hunting): authenticate will time out at 6471.537714s 2024-03-25T19:36:58.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.270+0000 7ff08cd9a640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7ff0880b1730 0x7ff088135b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:55132/0 (socket says 172.21.15.19:55132) 2024-03-25T19:36:58.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.270+0000 7ff08cd9a640 1 -- 172.21.15.19:0/4212546129 learned_addr learned my addr 172.21.15.19:0/4212546129 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:36:58.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.271+0000 7ff0857fa640 10 client.?.objecter ms_handle_connect 0x7ff0880b1730 2024-03-25T19:36:58.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.271+0000 7ff0857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:58.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.276+0000 7ff0857fa640 10 client.?.objecter ms_handle_connect 0x7ff08814be90 2024-03-25T19:36:58.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.276+0000 7ff0857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:58.317 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:21.047 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.323 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:21.053 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.343+0000 7ff0857fa640 10 client.?.objecter ms_handle_connect 0x7ff0880b4d60 2024-03-25T19:36:58.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.343+0000 7ff0857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:58.346 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:21.076 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.373 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:08:21.076 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.375+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106606583 0 0) 0x7ff088137f30 con 0x7ff0880b1730 2024-03-25T19:36:58.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.375+0000 7ff0857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.375+0000 7ff0857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.375+0000 7ff0857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.375+0000 7ff0857fa640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:36:58.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.375+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff06c004eb0 con 0x7ff0880b1730 2024-03-25T19:36:58.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.377+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477285734 0 0) 0x7ff088137b10 con 0x7ff0880b4d60 2024-03-25T19:36:58.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.377+0000 7ff0857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.377+0000 7ff0857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.377+0000 7ff0857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.377+0000 7ff0857fa640 10 monclient(hunting): my global_id is 4587 2024-03-25T19:36:58.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.377+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff088137f30 con 0x7ff0880b4d60 2024-03-25T19:36:58.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.382+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867578148 0 0) 0x7ff0881380c0 con 0x7ff08814be90 2024-03-25T19:36:58.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.382+0000 7ff0857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:58.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.382+0000 7ff0857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:58.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.382+0000 7ff0857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:58.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.382+0000 7ff0857fa640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:36:58.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.382+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff088137b10 con 0x7ff08814be90 2024-03-25T19:36:58.382 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:21.111 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.392 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:08:21.094 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.405 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:21.135 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.409+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3834873780 0 0) 0x7ff06c004eb0 con 0x7ff0880b1730 2024-03-25T19:36:58.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.409+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0881380c0 con 0x7ff0880b1730 2024-03-25T19:36:58.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.409+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff07c0018b0 con 0x7ff0880b1730 2024-03-25T19:36:58.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.409+0000 7ff0857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.409+0000 7ff0857fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:58.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.409+0000 7ff0857fa640 10 monclient(hunting): dump: 2024-03-25T19:36:58.408 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:58.408 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:58.408 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:58.408 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:58.409 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.409 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:58.409 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:58.409 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:58.409 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:58.409 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:58.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.409+0000 7ff0857fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:58.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.412+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1194097903 0 0) 0x7ff088137b10 con 0x7ff08814be90 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.412+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff06c004eb0 con 0x7ff08814be90 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.412+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff080001a60 con 0x7ff08814be90 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.412+0000 7ff0857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.412+0000 7ff0857fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.412+0000 7ff0857fa640 10 monclient(hunting): dump: 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:58.412 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:58.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.423+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3076298070 0 0) 0x7ff088137f30 con 0x7ff0880b4d60 2024-03-25T19:36:58.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.423+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff088137b10 con 0x7ff0880b4d60 2024-03-25T19:36:58.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.423+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff074001b80 con 0x7ff0880b4d60 2024-03-25T19:36:58.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.423+0000 7ff0857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.423+0000 7ff0857fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:58.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.423+0000 7ff0857fa640 10 monclient(hunting): dump: 2024-03-25T19:36:58.423 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:58.423 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:58.423 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:58.423 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:58.423 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.423 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:58.423 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:58.423 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:58.423 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:58.423 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:58.428 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:08:21.131 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3179208618 0 0) 0x7ff088137b10 con 0x7ff0880b4d60 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 >> v1:172.21.15.19:6790/0 conn(0x7ff08814be90 legacy=0x7ff08812f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 >> v1:172.21.15.188:6789/0 conn(0x7ff0880b1730 legacy=0x7ff088135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff0857fa640 10 monclient: _finish_hunting 0 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff0857fa640 1 monclient: found mon.a 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff0857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff0857fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0881367f0 con 0x7ff0880b4d60 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff0857fa640 10 monclient: _finish_auth 0 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff0857fa640 10 monclient: _check_auth_tickets 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff0857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff08effe640 5 monclient: authenticate success, global_id 4587 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff08effe640 10 monclient: _renew_subs 2024-03-25T19:36:58.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff08effe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff08effe640 1 -- 172.21.15.19:0/4212546129 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08813ea50 con 0x7ff0880b4d60 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff08effe640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff08effe640 10 monclient: _renew_subs 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff08effe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff08effe640 1 -- 172.21.15.19:0/4212546129 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff088145630 con 0x7ff0880b4d60 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.437+0000 7ff08effe640 1 librados: init done 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.438+0000 7ff08effe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.438+0000 7ff08effe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:58.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.438+0000 7ff08effe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:58.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.438+0000 7ff08effe640 1 -- 172.21.15.19:0/4212546129 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0881458e0 con 0x7ff0880b4d60 2024-03-25T19:36:58.447 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:21.177 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.666+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff074003960 con 0x7ff0880b4d60 2024-03-25T19:36:58.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.666+0000 7ff0857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:58.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.666+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff074001250 con 0x7ff0880b4d60 2024-03-25T19:36:58.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.666+0000 7ff0857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.666+0000 7ff0857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:58.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.666+0000 7ff0857fa640 10 monclient: dump: 2024-03-25T19:36:58.665 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:36:58.665 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:36:58.665 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:58.665 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:36:58.666 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.666 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:36:58.666 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:36:58.666 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:36:58.666 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:36:58.666 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:36:58.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.666+0000 7ff086ffd640 4 set_mon_vals no callback set 2024-03-25T19:36:58.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.750+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7ff074019ae0 con 0x7ff0880b4d60 2024-03-25T19:36:58.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.751+0000 7ff08cd9a640 10 monclient: get_auth_request con 0x7ff06c06e160 auth_method 0 2024-03-25T19:36:58.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.751+0000 7ff0857fa640 10 client.4587.objecter ms_handle_connect 0x7ff06c06e160 2024-03-25T19:36:58.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.758+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7ff0740536c0 con 0x7ff0880b4d60 2024-03-25T19:36:58.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.758+0000 7ff0857fa640 10 client.4587.objecter ms_dispatch 0x7ff088000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:58.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.758+0000 7ff0857fa640 3 client.4587.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:58.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.758+0000 7ff0857fa640 3 client.4587.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:58.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.758+0000 7ff0857fa640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-25T19:36:58.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.758+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff07404f1d0 con 0x7ff0880b4d60 2024-03-25T19:36:58.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.759+0000 7ff0857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:58.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:58.759+0000 7ff0857fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:59.015 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.016+0000 7ff08effe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:59.015 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.016+0000 7ff08effe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:59.015 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.016+0000 7ff08effe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:36:59.015 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.016+0000 7ff08effe640 1 -- 172.21.15.19:0/4212546129 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0881401f0 con 0x7ff0880b4d60 2024-03-25T19:36:59.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.038+0000 7ff0857fa640 1 -- 172.21.15.19:0/4212546129 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7ff074051100 con 0x7ff0880b4d60 2024-03-25T19:36:59.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.038+0000 7ff0857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:59.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.038+0000 7ff0857fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:59.037 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:36:59.037 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:59.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.040+0000 7ff08effe640 10 librados: watch_flush enter 2024-03-25T19:36:59.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.040+0000 7ff08effe640 10 librados: watch_flush exit 2024-03-25T19:36:59.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.040+0000 7ff08effe640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-25T19:36:59.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.040+0000 7ff08effe640 10 client.4587.objecter successfully canceled tick 2024-03-25T19:36:59.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.040+0000 7ff08effe640 1 -- 172.21.15.19:0/4212546129 >> v1:172.21.15.188:6800/2207506229 conn(0x7ff06c06e160 legacy=0x7ff06c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:59.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.040+0000 7ff08effe640 10 monclient: shutdown 2024-03-25T19:36:59.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.040+0000 7ff08effe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:59.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.040+0000 7ff08effe640 1 -- 172.21.15.19:0/4212546129 >> v1:172.21.15.19:6789/0 conn(0x7ff0880b4d60 legacy=0x7ff08812e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:59.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.040+0000 7ff08effe640 1 -- 172.21.15.19:0/4212546129 shutdown_connections 2024-03-25T19:36:59.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.040+0000 7ff08effe640 1 -- 172.21.15.19:0/4212546129 >> 172.21.15.19:0/4212546129 conn(0x7ff0880617a0 msgr2=0x7ff08813c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:59.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.041+0000 7ff08effe640 1 -- 172.21.15.19:0/4212546129 shutdown_connections 2024-03-25T19:36:59.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.041+0000 7ff08effe640 1 -- 172.21.15.19:0/4212546129 wait complete. 2024-03-25T19:36:59.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:36:59.041+0000 7ff08effe640 1 librados: shutdown 2024-03-25T19:36:59.051 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:05.857 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.973+0000 7f99af80e640 10 monclient: get_monmap_and_config 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.973+0000 7f99af80e640 10 monclient: build_initial_monmap 2024-03-25T19:37:05.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.973+0000 7f99af80e640 10 monclient: monmap: 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:37:05.973888+0000 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:37:05.973888+0000 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:37:05.973 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.974+0000 7f99af80e640 1 Processor -- start 2024-03-25T19:37:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.974+0000 7f99af80e640 1 -- start start 2024-03-25T19:37:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.974+0000 7f99af80e640 10 monclient: init 2024-03-25T19:37:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.975+0000 7f99af80e640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.975+0000 7f99af80e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.975+0000 7f99af80e640 10 monclient(hunting): picked mon.noname-a con 0x7f99a80a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.975+0000 7f99af80e640 10 monclient(hunting): picked mon.noname-c con 0x7f99a813f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.975+0000 7f99af80e640 10 monclient(hunting): picked mon.noname-b con 0x7f99a81430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.975+0000 7f99af80e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a8142520 con 0x7f99a80a98f0 2024-03-25T19:37:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.975+0000 7f99af80e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a80b28a0 con 0x7f99a813f710 2024-03-25T19:37:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.975+0000 7f99af80e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a80b2a30 con 0x7f99a81430c0 2024-03-25T19:37:05.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.975+0000 7f99af80e640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:05.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.975+0000 7f99af80e640 10 monclient(hunting): authenticate will time out at 6479.244649s 2024-03-25T19:37:05.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.976+0000 7f99acda9640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f99a813f710 0x7f99a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:43844/0 (socket says 172.21.15.19:43844) 2024-03-25T19:37:05.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:05.976+0000 7f99acda9640 1 -- 172.21.15.19:0/2291453175 learned_addr learned my addr 172.21.15.19:0/2291453175 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:06.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.048+0000 7f999ffff640 1 -- 172.21.15.19:0/2291453175 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512343587 0 0) 0x7f99a80b28a0 con 0x7f99a813f710 2024-03-25T19:37:06.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.048+0000 7f999ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:06.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.048+0000 7f999ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:06.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.048+0000 7f999ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:06.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.048+0000 7f999ffff640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:37:06.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.048+0000 7f999ffff640 1 -- 172.21.15.19:0/2291453175 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9994003520 con 0x7f99a813f710 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.064+0000 7f999ffff640 1 -- 172.21.15.19:0/2291453175 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325743152 0 0) 0x7f99a8142520 con 0x7f99a80a98f0 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.064+0000 7f999ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.064+0000 7f999ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.064+0000 7f999ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.064+0000 7f999ffff640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:37:06.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.064+0000 7f999ffff640 1 -- 172.21.15.19:0/2291453175 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a80b28a0 con 0x7f99a80a98f0 2024-03-25T19:37:06.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.065+0000 7f999ffff640 1 -- 172.21.15.19:0/2291453175 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183678078 0 0) 0x7f9994003520 con 0x7f99a813f710 2024-03-25T19:37:06.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.065+0000 7f999ffff640 1 -- 172.21.15.19:0/2291453175 >> v1:172.21.15.19:6790/0 conn(0x7f99a81430c0 legacy=0x7f99a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.065+0000 7f999ffff640 1 -- 172.21.15.19:0/2291453175 >> v1:172.21.15.19:6789/0 conn(0x7f99a80a98f0 legacy=0x7f99a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.065+0000 7f999ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:06.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.065+0000 7f999ffff640 1 monclient: found mon.noname-c 2024-03-25T19:37:06.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.065+0000 7f999ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:06.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.066+0000 7f999ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:37:06.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.066+0000 7f999ffff640 1 -- 172.21.15.19:0/2291453175 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a80b44c0 con 0x7f99a813f710 2024-03-25T19:37:06.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.066+0000 7f999ffff640 10 monclient: _finish_auth 0 2024-03-25T19:37:06.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.066+0000 7f999ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:06.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.066+0000 7f999ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.066+0000 7f99af80e640 5 monclient: authenticate success, global_id 4579 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.066+0000 7f999ffff640 1 -- 172.21.15.19:0/2291453175 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9998001690 con 0x7f99a813f710 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.066+0000 7f99af80e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.066+0000 7f999ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.066+0000 7f999ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.066+0000 7f999ffff640 10 monclient: dump: 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:06.066 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:06.067 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:06.067 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:06.067 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:06.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.066+0000 7f99af80e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:06.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.085+0000 7f999ffff640 1 -- 172.21.15.19:0/2291453175 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99980017f0 con 0x7f99a813f710 2024-03-25T19:37:06.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.085+0000 7f999ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:06.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.085+0000 7f999ffff640 1 -- 172.21.15.19:0/2291453175 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9998001c00 con 0x7f99a813f710 2024-03-25T19:37:06.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.085+0000 7f999ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:06.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.085+0000 7f999ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:06.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.085+0000 7f999ffff640 10 monclient: dump: 2024-03-25T19:37:06.085 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:06.085 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:06.085 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:06.085 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:06.085 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:06.085 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:06.085 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:06.085 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:06.085 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:06.085 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:06.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.085+0000 7f99af80e640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:06.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.085+0000 7f99af80e640 4 set_mon_vals no callback set 2024-03-25T19:37:06.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.086+0000 7f99af80e640 10 monclient: shutdown 2024-03-25T19:37:06.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.086+0000 7f99af80e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.086+0000 7f99af80e640 1 -- 172.21.15.19:0/2291453175 >> v1:172.21.15.188:6789/0 conn(0x7f99a813f710 legacy=0x7f99a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.086+0000 7f99af80e640 1 -- 172.21.15.19:0/2291453175 shutdown_connections 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.086+0000 7f99af80e640 1 -- 172.21.15.19:0/2291453175 >> 172.21.15.19:0/2291453175 conn(0x7f99a80b7ea0 msgr2=0x7f99a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.086+0000 7f99af80e640 1 -- 172.21.15.19:0/2291453175 shutdown_connections 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.086+0000 7f99af80e640 1 -- 172.21.15.19:0/2291453175 wait complete. 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.087+0000 7f99af80e640 10 monclient: build_initial_monmap 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.087+0000 7f99af80e640 10 monclient: monmap: 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:37:06.088 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.087+0000 7f99af80e640 1 librados: starting msgr at 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.087+0000 7f99af80e640 1 librados: starting objecter 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.087+0000 7f99af80e640 1 Processor -- start 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.087+0000 7f99af80e640 1 -- start start 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.087+0000 7f99af80e640 1 librados: setting wanted keys 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.087+0000 7f99af80e640 1 librados: calling monclient init 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.087+0000 7f99af80e640 10 monclient: init 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.088+0000 7f99af80e640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.088+0000 7f99af80e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.088+0000 7f99af80e640 10 monclient(hunting): picked mon.noname-b con 0x7f99a80a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:06.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.088+0000 7f99af80e640 10 monclient(hunting): picked mon.noname-c con 0x7f99a813f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:06.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.088+0000 7f99af80e640 10 monclient(hunting): picked mon.noname-a con 0x7f99a81430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:06.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.088+0000 7f99af80e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a8138a70 con 0x7f99a81430c0 2024-03-25T19:37:06.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.088+0000 7f99af80e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a80aba50 con 0x7f99a80a98f0 2024-03-25T19:37:06.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.088+0000 7f99af80e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a81d77a0 con 0x7f99a813f710 2024-03-25T19:37:06.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.088+0000 7f99af80e640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:06.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.088+0000 7f99af80e640 10 monclient(hunting): authenticate will time out at 6479.357422s 2024-03-25T19:37:06.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.097+0000 7f99ad5aa640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f99a80a98f0 0x7f99a8138360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:43884/0 (socket says 172.21.15.19:43884) 2024-03-25T19:37:06.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.097+0000 7f99ad5aa640 1 -- 172.21.15.19:0/2746735351 learned_addr learned my addr 172.21.15.19:0/2746735351 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:06.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.098+0000 7f999dffb640 10 client.?.objecter ms_handle_connect 0x7f99a813f710 2024-03-25T19:37:06.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.098+0000 7f999dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:06.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.098+0000 7f999dffb640 10 client.?.objecter ms_handle_connect 0x7f99a80a98f0 2024-03-25T19:37:06.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.098+0000 7f999dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:06.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.110+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195512766 0 0) 0x7f99a80aba50 con 0x7f99a80a98f0 2024-03-25T19:37:06.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.110+0000 7f999dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:06.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.110+0000 7f999dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:06.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.110+0000 7f999dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:06.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.110+0000 7f999dffb640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:37:06.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.110+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f998c004eb0 con 0x7f99a80a98f0 2024-03-25T19:37:06.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.117+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790763172 0 0) 0x7f99a81d77a0 con 0x7f99a813f710 2024-03-25T19:37:06.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.117+0000 7f999dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:06.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.117+0000 7f999dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:06.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.117+0000 7f999dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:06.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.117+0000 7f999dffb640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:37:06.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.117+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a80aba50 con 0x7f99a813f710 2024-03-25T19:37:06.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.125+0000 7f999dffb640 10 client.?.objecter ms_handle_connect 0x7f99a81430c0 2024-03-25T19:37:06.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.125+0000 7f999dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:06.134 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.135+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1412545579 0 0) 0x7f99a80aba50 con 0x7f99a813f710 2024-03-25T19:37:06.134 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.135+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99a81d77a0 con 0x7f99a813f710 2024-03-25T19:37:06.134 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.135+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9998001bc0 con 0x7f99a813f710 2024-03-25T19:37:06.134 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.135+0000 7f999dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:06.134 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.135+0000 7f999dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:06.134 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.135+0000 7f999dffb640 10 monclient(hunting): dump: 2024-03-25T19:37:06.134 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:06.134 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:06.134 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:06.135 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:06.135 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:06.135 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:06.135 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:06.135 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:06.135 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:06.135 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:06.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.135+0000 7f999dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:06.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.139+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1769903437 0 0) 0x7f998c004eb0 con 0x7f99a80a98f0 2024-03-25T19:37:06.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.139+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99a80aba50 con 0x7f99a80a98f0 2024-03-25T19:37:06.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.139+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f99a4001a10 con 0x7f99a80a98f0 2024-03-25T19:37:06.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.139+0000 7f999dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:06.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.139+0000 7f999dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:06.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.139+0000 7f999dffb640 10 monclient(hunting): dump: 2024-03-25T19:37:06.139 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:06.139 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:06.139 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:06.139 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:06.139 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:06.139 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:06.139 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:06.139 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:06.139 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:06.139 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:06.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.144+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488107532 0 0) 0x7f99a8138a70 con 0x7f99a81430c0 2024-03-25T19:37:06.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.144+0000 7f999dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:06.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.144+0000 7f999dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:06.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.144+0000 7f999dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:06.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.144+0000 7f999dffb640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:37:06.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.144+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f998c004eb0 con 0x7f99a81430c0 2024-03-25T19:37:06.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 819843219 0 0) 0x7f99a81d77a0 con 0x7f99a813f710 2024-03-25T19:37:06.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 >> v1:172.21.15.188:6789/0 conn(0x7f99a80a98f0 legacy=0x7f99a8138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 >> v1:172.21.15.19:6789/0 conn(0x7f99a81430c0 legacy=0x7f99a81d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f999dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f999dffb640 1 monclient: found mon.c 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f999dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f999dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a81d83d0 con 0x7f99a813f710 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f999dffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f999dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f999dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f99af80e640 5 monclient: authenticate success, global_id 4559 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f99af80e640 10 monclient: _renew_subs 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f99af80e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f99af80e640 1 -- 172.21.15.19:0/2746735351 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a81d41e0 con 0x7f99a813f710 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f99af80e640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:06.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f99af80e640 10 monclient: _renew_subs 2024-03-25T19:37:06.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f99af80e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:06.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f99af80e640 1 -- 172.21.15.19:0/2746735351 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a81d5110 con 0x7f99a813f710 2024-03-25T19:37:06.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.155+0000 7f99af80e640 1 librados: init done 2024-03-25T19:37:06.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.156+0000 7f99af80e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:06.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.156+0000 7f99af80e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:06.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.156+0000 7f99af80e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:06.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.156+0000 7f99af80e640 1 -- 172.21.15.19:0/2746735351 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a814f320 con 0x7f99a813f710 2024-03-25T19:37:06.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.498+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9998004a90 con 0x7f99a813f710 2024-03-25T19:37:06.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.498+0000 7f999dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:06.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.498+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9998004da0 con 0x7f99a813f710 2024-03-25T19:37:06.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.498+0000 7f999dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:06.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.498+0000 7f999dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:06.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.498+0000 7f999dffb640 10 monclient: dump: 2024-03-25T19:37:06.498 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:06.498 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:06.498 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:06.498 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:06.498 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:06.498 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:06.499 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:06.499 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:06.499 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:06.499 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:06.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.498+0000 7f999f7fe640 4 set_mon_vals no callback set 2024-03-25T19:37:06.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.499+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f999801ad70 con 0x7f99a813f710 2024-03-25T19:37:06.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.499+0000 7f99ad5aa640 10 monclient: get_auth_request con 0x7f998c06df50 auth_method 0 2024-03-25T19:37:06.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.499+0000 7f999dffb640 10 client.4559.objecter ms_handle_connect 0x7f998c06df50 2024-03-25T19:37:06.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.507+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f9998054880 con 0x7f99a813f710 2024-03-25T19:37:06.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.507+0000 7f999dffb640 10 client.4559.objecter ms_dispatch 0x7f99a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:06.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.507+0000 7f999dffb640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:06.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.507+0000 7f999dffb640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:06.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.507+0000 7f999dffb640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-25T19:37:06.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.507+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9998050390 con 0x7f99a813f710 2024-03-25T19:37:06.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.507+0000 7f999dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:06.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.507+0000 7f999dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:06.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.782+0000 7f99af80e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:06.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.782+0000 7f99af80e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:06.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.782+0000 7f99af80e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:06.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.782+0000 7f99af80e640 1 -- 172.21.15.19:0/2746735351 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99a81d7930 con 0x7f99a813f710 2024-03-25T19:37:06.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.827+0000 7f999dffb640 1 -- 172.21.15.19:0/2746735351 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f99980522c0 con 0x7f99a813f710 2024-03-25T19:37:06.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.827+0000 7f999dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:06.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.827+0000 7f999dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:06.827 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:37:06.827 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:06.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.830+0000 7f99af80e640 10 librados: watch_flush enter 2024-03-25T19:37:06.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.830+0000 7f99af80e640 10 librados: watch_flush exit 2024-03-25T19:37:06.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.830+0000 7f99af80e640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-25T19:37:06.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.830+0000 7f99af80e640 10 client.4559.objecter successfully canceled tick 2024-03-25T19:37:06.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.830+0000 7f99af80e640 1 -- 172.21.15.19:0/2746735351 >> v1:172.21.15.188:6800/2207506229 conn(0x7f998c06df50 legacy=0x7f998c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.830+0000 7f99af80e640 10 monclient: shutdown 2024-03-25T19:37:06.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.830+0000 7f99af80e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:06.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.830+0000 7f99af80e640 1 -- 172.21.15.19:0/2746735351 >> v1:172.21.15.19:6790/0 conn(0x7f99a813f710 legacy=0x7f99a81d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.830+0000 7f99af80e640 1 -- 172.21.15.19:0/2746735351 shutdown_connections 2024-03-25T19:37:06.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.830+0000 7f99af80e640 1 -- 172.21.15.19:0/2746735351 >> 172.21.15.19:0/2746735351 conn(0x7f99a80b7ea0 msgr2=0x7f99a813ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:06.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.830+0000 7f99af80e640 1 -- 172.21.15.19:0/2746735351 shutdown_connections 2024-03-25T19:37:06.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.830+0000 7f99af80e640 1 -- 172.21.15.19:0/2746735351 wait complete. 2024-03-25T19:37:06.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:06.831+0000 7f99af80e640 1 librados: shutdown 2024-03-25T19:37:06.842 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:13.647 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:13.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.759+0000 7fa539f2e640 10 monclient: get_monmap_and_config 2024-03-25T19:37:13.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.759+0000 7fa539f2e640 10 monclient: build_initial_monmap 2024-03-25T19:37:13.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.759+0000 7fa539f2e640 10 monclient: monmap: 2024-03-25T19:37:13.760 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:13.760 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:13.760 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:37:13.761170+0000 2024-03-25T19:37:13.760 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:37:13.761170+0000 2024-03-25T19:37:13.760 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:13.760 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:13.760 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:13.761 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:37:13.761 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:37:13.761 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:13.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.760+0000 7fa539f2e640 1 Processor -- start 2024-03-25T19:37:13.761 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.760+0000 7fa539f2e640 1 -- start start 2024-03-25T19:37:13.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.760+0000 7fa539f2e640 10 monclient: init 2024-03-25T19:37:13.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.761+0000 7fa539f2e640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:13.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.761+0000 7fa539f2e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:13.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.761+0000 7fa539f2e640 10 monclient(hunting): picked mon.noname-b con 0x7fa5340a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:13.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.761+0000 7fa539f2e640 10 monclient(hunting): picked mon.noname-a con 0x7fa53413f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:13.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.761+0000 7fa539f2e640 10 monclient(hunting): picked mon.noname-c con 0x7fa5341430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:13.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.761+0000 7fa539f2e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa534142520 con 0x7fa53413f710 2024-03-25T19:37:13.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.761+0000 7fa539f2e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5340b28a0 con 0x7fa5341430c0 2024-03-25T19:37:13.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.761+0000 7fa539f2e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5340b2a30 con 0x7fa5340a98f0 2024-03-25T19:37:13.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.761+0000 7fa539f2e640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:13.762 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.761+0000 7fa539f2e640 10 monclient(hunting): authenticate will time out at 6487.031472s 2024-03-25T19:37:13.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.770+0000 7fa5337fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fa5340a98f0 0x7fa5340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:46496/0 (socket says 172.21.15.19:46496) 2024-03-25T19:37:13.770 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.770+0000 7fa5337fe640 1 -- 172.21.15.19:0/975185279 learned_addr learned my addr 172.21.15.19:0/975185279 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:13.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.784+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552533669 0 0) 0x7fa5340b2a30 con 0x7fa5340a98f0 2024-03-25T19:37:13.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.784+0000 7fa5327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.784+0000 7fa5327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.784+0000 7fa5327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.784+0000 7fa5327fc640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:37:13.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.784+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa51c003520 con 0x7fa5340a98f0 2024-03-25T19:37:13.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.796+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003877836 0 0) 0x7fa5340b28a0 con 0x7fa5341430c0 2024-03-25T19:37:13.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.796+0000 7fa5327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.796+0000 7fa5327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.796+0000 7fa5327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.796+0000 7fa5327fc640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:37:13.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.796+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5340b2a30 con 0x7fa5341430c0 2024-03-25T19:37:13.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.808+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781150932 0 0) 0x7fa534142520 con 0x7fa53413f710 2024-03-25T19:37:13.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.808+0000 7fa5327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.808+0000 7fa5327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.808+0000 7fa5327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.808+0000 7fa5327fc640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:37:13.808 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.808+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5340b28a0 con 0x7fa53413f710 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3034914238 0 0) 0x7fa51c003520 con 0x7fa5340a98f0 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 >> v1:172.21.15.188:6789/0 conn(0x7fa5341430c0 legacy=0x7fa53414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 >> v1:172.21.15.19:6789/0 conn(0x7fa53413f710 legacy=0x7fa5341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 1 monclient: found mon.noname-b 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5340b44c0 con 0x7fa5340a98f0 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa524001690 con 0x7fa5340a98f0 2024-03-25T19:37:13.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa5327fc640 10 monclient: dump: 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa539f2e640 5 monclient: authenticate success, global_id 4562 2024-03-25T19:37:13.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.810+0000 7fa539f2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:13.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.832+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5240017f0 con 0x7fa5340a98f0 2024-03-25T19:37:13.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.832+0000 7fa5327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:13.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.832+0000 7fa5327fc640 1 -- 172.21.15.19:0/975185279 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa524001c00 con 0x7fa5340a98f0 2024-03-25T19:37:13.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.832+0000 7fa5327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.832+0000 7fa5327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:13.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.832+0000 7fa5327fc640 10 monclient: dump: 2024-03-25T19:37:13.832 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:13.832 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:13.833 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:13.833 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:13.833 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.833 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:13.833 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:13.833 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:13.833 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:13.833 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:13.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.832+0000 7fa539f2e640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:13.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.832+0000 7fa539f2e640 4 set_mon_vals no callback set 2024-03-25T19:37:13.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.833+0000 7fa539f2e640 10 monclient: shutdown 2024-03-25T19:37:13.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.833+0000 7fa539f2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:13.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.833+0000 7fa539f2e640 1 -- 172.21.15.19:0/975185279 >> v1:172.21.15.19:6790/0 conn(0x7fa5340a98f0 legacy=0x7fa5340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.833+0000 7fa539f2e640 1 -- 172.21.15.19:0/975185279 shutdown_connections 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.833+0000 7fa539f2e640 1 -- 172.21.15.19:0/975185279 >> 172.21.15.19:0/975185279 conn(0x7fa5340b7ea0 msgr2=0x7fa5340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.833+0000 7fa539f2e640 1 -- 172.21.15.19:0/975185279 shutdown_connections 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.833+0000 7fa539f2e640 1 -- 172.21.15.19:0/975185279 wait complete. 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.833+0000 7fa539f2e640 10 monclient: build_initial_monmap 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.833+0000 7fa539f2e640 10 monclient: monmap: 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:37:13.836 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.834+0000 7fa539f2e640 1 librados: starting msgr at 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.834+0000 7fa539f2e640 1 librados: starting objecter 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.834+0000 7fa539f2e640 1 Processor -- start 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.834+0000 7fa539f2e640 1 -- start start 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.834+0000 7fa539f2e640 1 librados: setting wanted keys 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.834+0000 7fa539f2e640 1 librados: calling monclient init 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.834+0000 7fa539f2e640 10 monclient: init 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.835+0000 7fa539f2e640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.835+0000 7fa539f2e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.835+0000 7fa539f2e640 10 monclient(hunting): picked mon.noname-a con 0x7fa5340a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.835+0000 7fa539f2e640 10 monclient(hunting): picked mon.noname-c con 0x7fa53413f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.835+0000 7fa539f2e640 10 monclient(hunting): picked mon.noname-b con 0x7fa5341430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.835+0000 7fa539f2e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa534135610 con 0x7fa5340a98f0 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.835+0000 7fa539f2e640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa534137230 con 0x7fa5341430c0 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.835+0000 7fa539f2e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa534135cf0 con 0x7fa53413f710 2024-03-25T19:37:13.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.835+0000 7fa539f2e640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:13.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.835+0000 7fa539f2e640 10 monclient(hunting): authenticate will time out at 6487.105150s 2024-03-25T19:37:13.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.843+0000 7fa533fff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fa5341430c0 0x7fa5341e2ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:39038/0 (socket says 172.21.15.19:39038) 2024-03-25T19:37:13.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.843+0000 7fa533fff640 1 -- 172.21.15.19:0/3065399179 learned_addr learned my addr 172.21.15.19:0/3065399179 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:13.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.845+0000 7fa51bfff640 10 client.?.objecter ms_handle_connect 0x7fa5341430c0 2024-03-25T19:37:13.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.845+0000 7fa51bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:13.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.848+0000 7fa51bfff640 10 client.?.objecter ms_handle_connect 0x7fa5340a98f0 2024-03-25T19:37:13.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.848+0000 7fa51bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:13.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.857+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231874851 0 0) 0x7fa534135610 con 0x7fa5340a98f0 2024-03-25T19:37:13.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.857+0000 7fa51bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.857+0000 7fa51bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.857+0000 7fa51bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.857+0000 7fa51bfff640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:37:13.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.857+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa520004eb0 con 0x7fa5340a98f0 2024-03-25T19:37:13.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.859+0000 7fa51bfff640 10 client.?.objecter ms_handle_connect 0x7fa53413f710 2024-03-25T19:37:13.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.859+0000 7fa51bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:13.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.863+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833094952 0 0) 0x7fa534137230 con 0x7fa5341430c0 2024-03-25T19:37:13.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.863+0000 7fa51bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.863+0000 7fa51bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.863+0000 7fa51bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.863+0000 7fa51bfff640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:37:13.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.863+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa534135610 con 0x7fa5341430c0 2024-03-25T19:37:13.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.868+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1894120660 0 0) 0x7fa520004eb0 con 0x7fa5340a98f0 2024-03-25T19:37:13.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.868+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa534137230 con 0x7fa5340a98f0 2024-03-25T19:37:13.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.868+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa524001b00 con 0x7fa5340a98f0 2024-03-25T19:37:13.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.868+0000 7fa51bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.868+0000 7fa51bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:13.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.868+0000 7fa51bfff640 10 monclient(hunting): dump: 2024-03-25T19:37:13.869 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:13.869 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:13.869 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:13.869 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:13.869 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.869 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:13.869 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:13.870 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:13.870 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:13.870 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:13.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.868+0000 7fa51bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:13.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4004090432 0 0) 0x7fa534137230 con 0x7fa5340a98f0 2024-03-25T19:37:13.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 >> v1:172.21.15.19:6790/0 conn(0x7fa53413f710 legacy=0x7fa5341dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 >> v1:172.21.15.188:6789/0 conn(0x7fa5341430c0 legacy=0x7fa5341e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa51bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:13.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa51bfff640 1 monclient: found mon.a 2024-03-25T19:37:13.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa51bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:13.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa51bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:37:13.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa534135e80 con 0x7fa5340a98f0 2024-03-25T19:37:13.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa51bfff640 10 monclient: _finish_auth 0 2024-03-25T19:37:13.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa51bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:13.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa51bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:13.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa539f2e640 5 monclient: authenticate success, global_id 4599 2024-03-25T19:37:13.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa539f2e640 10 monclient: _renew_subs 2024-03-25T19:37:13.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa539f2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:37:13.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa539f2e640 1 -- 172.21.15.19:0/3065399179 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa534131660 con 0x7fa5340a98f0 2024-03-25T19:37:13.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa539f2e640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:13.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa539f2e640 10 monclient: _renew_subs 2024-03-25T19:37:13.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa539f2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:37:13.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa539f2e640 1 -- 172.21.15.19:0/3065399179 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5340b7960 con 0x7fa5340a98f0 2024-03-25T19:37:13.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.874+0000 7fa539f2e640 1 librados: init done 2024-03-25T19:37:13.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.875+0000 7fa539f2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:13.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.875+0000 7fa539f2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:13.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.875+0000 7fa539f2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:37:13.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:13.875+0000 7fa539f2e640 1 -- 172.21.15.19:0/3065399179 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5340627b0 con 0x7fa5340a98f0 2024-03-25T19:37:14.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.150+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa524001220 con 0x7fa5340a98f0 2024-03-25T19:37:14.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.150+0000 7fa51bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:14.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.150+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa524004c30 con 0x7fa5340a98f0 2024-03-25T19:37:14.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.150+0000 7fa51bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.150+0000 7fa51bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.150+0000 7fa51bfff640 10 monclient: dump: 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:14.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.150+0000 7fa531ffb640 4 set_mon_vals no callback set 2024-03-25T19:37:14.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.151+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fa52401ac40 con 0x7fa5340a98f0 2024-03-25T19:37:14.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.151+0000 7fa532ffd640 10 monclient: get_auth_request con 0x7fa52006dd10 auth_method 0 2024-03-25T19:37:14.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.152+0000 7fa51bfff640 10 client.4599.objecter ms_handle_connect 0x7fa52006dd10 2024-03-25T19:37:14.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.158+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fa524054840 con 0x7fa5340a98f0 2024-03-25T19:37:14.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.158+0000 7fa51bfff640 10 client.4599.objecter ms_dispatch 0x7fa534000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:14.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.158+0000 7fa51bfff640 3 client.4599.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:14.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.158+0000 7fa51bfff640 3 client.4599.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:14.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.159+0000 7fa51bfff640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-25T19:37:14.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.159+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa524050180 con 0x7fa5340a98f0 2024-03-25T19:37:14.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.159+0000 7fa51bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:14.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.159+0000 7fa51bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:14.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.422+0000 7fa539f2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:14.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.422+0000 7fa539f2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:14.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.422+0000 7fa539f2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:37:14.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.422+0000 7fa539f2e640 1 -- 172.21.15.19:0/3065399179 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5341351c0 con 0x7fa5340a98f0 2024-03-25T19:37:14.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.451+0000 7fa51bfff640 1 -- 172.21.15.19:0/3065399179 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fa524052280 con 0x7fa5340a98f0 2024-03-25T19:37:14.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.451+0000 7fa51bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:14.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.451+0000 7fa51bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:14.451 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:37:14.451 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:14.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.453+0000 7fa539f2e640 10 librados: watch_flush enter 2024-03-25T19:37:14.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.453+0000 7fa539f2e640 10 librados: watch_flush exit 2024-03-25T19:37:14.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.453+0000 7fa539f2e640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-25T19:37:14.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.453+0000 7fa539f2e640 10 client.4599.objecter successfully canceled tick 2024-03-25T19:37:14.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.453+0000 7fa539f2e640 1 -- 172.21.15.19:0/3065399179 >> v1:172.21.15.188:6800/2207506229 conn(0x7fa52006dd10 legacy=0x7fa5200701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:14.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.453+0000 7fa539f2e640 10 monclient: shutdown 2024-03-25T19:37:14.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.453+0000 7fa539f2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:14.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.453+0000 7fa539f2e640 1 -- 172.21.15.19:0/3065399179 >> v1:172.21.15.19:6789/0 conn(0x7fa5340a98f0 legacy=0x7fa534130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:14.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.453+0000 7fa539f2e640 1 -- 172.21.15.19:0/3065399179 shutdown_connections 2024-03-25T19:37:14.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.453+0000 7fa539f2e640 1 -- 172.21.15.19:0/3065399179 >> 172.21.15.19:0/3065399179 conn(0x7fa5340b7ea0 msgr2=0x7fa534074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:14.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.454+0000 7fa539f2e640 1 -- 172.21.15.19:0/3065399179 shutdown_connections 2024-03-25T19:37:14.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.454+0000 7fa539f2e640 1 -- 172.21.15.19:0/3065399179 wait complete. 2024-03-25T19:37:14.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:14.454+0000 7fa539f2e640 1 librados: shutdown 2024-03-25T19:37:14.466 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:18.205 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:40.934 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.256 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:40.985 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.280 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:41.010 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.321 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:08:41.024 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.331 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:08:41.034 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.331 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:41.061 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.346 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:41.075 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.370 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:08:41.073 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.375 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:41.104 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:21.273 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:21.394 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.393+0000 7f107d547640 10 monclient: get_monmap_and_config 2024-03-25T19:37:21.394 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.393+0000 7f107d547640 10 monclient: build_initial_monmap 2024-03-25T19:37:21.394 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.393+0000 7f107d547640 10 monclient: monmap: 2024-03-25T19:37:21.394 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:21.394 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:21.394 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:37:21.395197+0000 2024-03-25T19:37:21.394 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:37:21.395197+0000 2024-03-25T19:37:21.394 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:21.394 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:21.394 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:21.394 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:37:21.395 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:37:21.395 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:21.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.394+0000 7f107d547640 1 Processor -- start 2024-03-25T19:37:21.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.395+0000 7f107d547640 1 -- start start 2024-03-25T19:37:21.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.395+0000 7f107d547640 10 monclient: init 2024-03-25T19:37:21.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.395+0000 7f107d547640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:21.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.395+0000 7f107d547640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:21.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.395+0000 7f107d547640 10 monclient(hunting): picked mon.noname-a con 0x7f10780b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:21.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.395+0000 7f107d547640 10 monclient(hunting): picked mon.noname-c con 0x7f10780b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:21.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.395+0000 7f107d547640 10 monclient(hunting): picked mon.noname-b con 0x7f107814be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:21.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.395+0000 7f107d547640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10780af940 con 0x7f10780b4d60 2024-03-25T19:37:21.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.395+0000 7f107d547640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10780b0a70 con 0x7f10780b1730 2024-03-25T19:37:21.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.395+0000 7f107d547640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10780b0d50 con 0x7f107814be90 2024-03-25T19:37:21.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.395+0000 7f107d547640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:21.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.395+0000 7f107d547640 10 monclient(hunting): authenticate will time out at 6494.665763s 2024-03-25T19:37:21.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.401+0000 7f10777fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f107814be90 0x7f107814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:43070/0 (socket says 172.21.15.19:43070) 2024-03-25T19:37:21.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.401+0000 7f10777fe640 1 -- 172.21.15.19:0/3419583865 learned_addr learned my addr 172.21.15.19:0/3419583865 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:21.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.414+0000 7f1075ffb640 1 -- 172.21.15.19:0/3419583865 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846809322 0 0) 0x7f10780b0a70 con 0x7f10780b1730 2024-03-25T19:37:21.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.414+0000 7f1075ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.414+0000 7f1075ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.414+0000 7f1075ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.414+0000 7f1075ffb640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:37:21.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.414+0000 7f1075ffb640 1 -- 172.21.15.19:0/3419583865 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1060003520 con 0x7f10780b1730 2024-03-25T19:37:21.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 1 -- 172.21.15.19:0/3419583865 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3858159377 0 0) 0x7f1060003520 con 0x7f10780b1730 2024-03-25T19:37:21.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 1 -- 172.21.15.19:0/3419583865 >> v1:172.21.15.19:6790/0 conn(0x7f107814be90 legacy=0x7f107814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 1 -- 172.21.15.19:0/3419583865 >> v1:172.21.15.19:6789/0 conn(0x7f10780b4d60 legacy=0x7f107813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 1 monclient: found mon.noname-c 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 1 -- 172.21.15.19:0/3419583865 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10780b1030 con 0x7f10780b1730 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 1 -- 172.21.15.19:0/3419583865 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f106c001690 con 0x7f10780b1730 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f107d547640 5 monclient: authenticate success, global_id 4591 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f107d547640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f1075ffb640 10 monclient: dump: 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:21.423 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:21.424 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:21.424 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:21.424 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.424 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:21.424 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:21.424 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:21.424 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:21.424 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:21.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.422+0000 7f107d547640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:21.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.432+0000 7f1075ffb640 1 -- 172.21.15.19:0/3419583865 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f106c0017f0 con 0x7f10780b1730 2024-03-25T19:37:21.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.432+0000 7f1075ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:21.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.432+0000 7f1075ffb640 1 -- 172.21.15.19:0/3419583865 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f106c001c00 con 0x7f10780b1730 2024-03-25T19:37:21.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.432+0000 7f107d547640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.432+0000 7f107d547640 4 set_mon_vals no callback set 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.432+0000 7f1075ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.432+0000 7f1075ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.432+0000 7f1075ffb640 10 monclient: dump: 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:21.433 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:21.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.433+0000 7f107d547640 10 monclient: shutdown 2024-03-25T19:37:21.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.433+0000 7f107d547640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:21.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.433+0000 7f107d547640 1 -- 172.21.15.19:0/3419583865 >> v1:172.21.15.188:6789/0 conn(0x7f10780b1730 legacy=0x7f10780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.433+0000 7f107d547640 1 -- 172.21.15.19:0/3419583865 shutdown_connections 2024-03-25T19:37:21.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.433+0000 7f107d547640 1 -- 172.21.15.19:0/3419583865 >> 172.21.15.19:0/3419583865 conn(0x7f10780617a0 msgr2=0x7f1078061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:21.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.433+0000 7f107d547640 1 -- 172.21.15.19:0/3419583865 shutdown_connections 2024-03-25T19:37:21.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.433+0000 7f107d547640 1 -- 172.21.15.19:0/3419583865 wait complete. 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.434+0000 7f107d547640 10 monclient: build_initial_monmap 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.434+0000 7f107d547640 10 monclient: monmap: 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:21.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.434+0000 7f107d547640 1 librados: starting msgr at 2024-03-25T19:37:21.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.434+0000 7f107d547640 1 librados: starting objecter 2024-03-25T19:37:21.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.434+0000 7f107d547640 1 Processor -- start 2024-03-25T19:37:21.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.434+0000 7f107d547640 1 -- start start 2024-03-25T19:37:21.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.434+0000 7f107d547640 1 librados: setting wanted keys 2024-03-25T19:37:21.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.434+0000 7f107d547640 1 librados: calling monclient init 2024-03-25T19:37:21.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.434+0000 7f107d547640 10 monclient: init 2024-03-25T19:37:21.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.435+0000 7f107d547640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:21.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.435+0000 7f107d547640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:21.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.435+0000 7f107d547640 10 monclient(hunting): picked mon.noname-c con 0x7f10780b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:21.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.435+0000 7f107d547640 10 monclient(hunting): picked mon.noname-a con 0x7f10780b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:21.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.435+0000 7f107d547640 10 monclient(hunting): picked mon.noname-b con 0x7f107814be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:21.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.435+0000 7f107d547640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10781dac30 con 0x7f10780b4d60 2024-03-25T19:37:21.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.435+0000 7f107d547640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10781dbab0 con 0x7f107814be90 2024-03-25T19:37:21.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.435+0000 7f107d547640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107813a640 con 0x7f10780b1730 2024-03-25T19:37:21.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.435+0000 7f107d547640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:21.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.435+0000 7f107d547640 10 monclient(hunting): authenticate will time out at 6494.705634s 2024-03-25T19:37:21.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.437+0000 7f10777fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f107814be90 0x7f10781e2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:60688/0 (socket says 172.21.15.19:60688) 2024-03-25T19:37:21.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.437+0000 7f10777fe640 1 -- 172.21.15.19:0/1759746974 learned_addr learned my addr 172.21.15.19:0/1759746974 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:21.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.438+0000 7f105f7fe640 10 client.?.objecter ms_handle_connect 0x7f107814be90 2024-03-25T19:37:21.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.438+0000 7f105f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:21.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.447+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274575828 0 0) 0x7f10781dbab0 con 0x7f107814be90 2024-03-25T19:37:21.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.447+0000 7f105f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.447+0000 7f105f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.447+0000 7f105f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.447+0000 7f105f7fe640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:37:21.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.447+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1064004eb0 con 0x7f107814be90 2024-03-25T19:37:21.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.457+0000 7f10767fc640 1 -- 172.21.15.19:0/1759746974 >> v1:172.21.15.19:6789/0 conn(0x7f10780b4d60 legacy=0x7f107813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T19:37:21.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.457+0000 7f10767fc640 1 -- 172.21.15.19:0/1759746974 >> v1:172.21.15.19:6789/0 conn(0x7f10780b4d60 legacy=0x7f107813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:37:21.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.457+0000 7f10767fc640 1 --1- 172.21.15.19:0/1759746974 >> v1:172.21.15.19:6789/0 conn(0x7f10780b4d60 0x7f107813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-25T19:37:21.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.457+0000 7f10767fc640 1 --1- 172.21.15.19:0/1759746974 >> v1:172.21.15.19:6789/0 conn(0x7f10780b4d60 0x7f107813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T19:37:21.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.457+0000 7f105f7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.19:6789/0 2024-03-25T19:37:21.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.466+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1230413818 0 0) 0x7f1064004eb0 con 0x7f107814be90 2024-03-25T19:37:21.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.466+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10781dbab0 con 0x7f107814be90 2024-03-25T19:37:21.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.466+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1070001b10 con 0x7f107814be90 2024-03-25T19:37:21.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.466+0000 7f105f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.466+0000 7f105f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:21.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.466+0000 7f105f7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:21.466 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:21.466 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:21.466 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:21.466 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:21.467 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.467 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:21.467 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:21.467 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:21.467 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:21.467 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:21.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.466+0000 7f105f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:21.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.468+0000 7f105f7fe640 10 client.?.objecter ms_handle_connect 0x7f10780b1730 2024-03-25T19:37:21.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.468+0000 7f105f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:21.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.482+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318699868 0 0) 0x7f107813a640 con 0x7f10780b1730 2024-03-25T19:37:21.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.482+0000 7f105f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.482+0000 7f105f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.482+0000 7f105f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.482+0000 7f105f7fe640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:37:21.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.482+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1064004eb0 con 0x7f10780b1730 2024-03-25T19:37:21.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2011855624 0 0) 0x7f10781dbab0 con 0x7f107814be90 2024-03-25T19:37:21.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 >> v1:172.21.15.19:6790/0 conn(0x7f10780b1730 legacy=0x7f10781db3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 >> v1:172.21.15.19:6789/0 conn(0x7f10780b4d60 legacy=0x7f107813b5f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:37:21.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f105f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:21.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f105f7fe640 1 monclient: found mon.b 2024-03-25T19:37:21.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f105f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:21.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f105f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:37:21.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10781dae80 con 0x7f107814be90 2024-03-25T19:37:21.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f105f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:21.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f105f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:21.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f105f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:21.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f107d547640 5 monclient: authenticate success, global_id 4594 2024-03-25T19:37:21.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f107d547640 10 monclient: _renew_subs 2024-03-25T19:37:21.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f107d547640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:37:21.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f107d547640 1 -- 172.21.15.19:0/1759746974 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1078143230 con 0x7f107814be90 2024-03-25T19:37:21.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f107d547640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:21.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f107d547640 10 monclient: _renew_subs 2024-03-25T19:37:21.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f107d547640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:37:21.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f107d547640 1 -- 172.21.15.19:0/1759746974 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1078142a70 con 0x7f107814be90 2024-03-25T19:37:21.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.489+0000 7f107d547640 1 librados: init done 2024-03-25T19:37:21.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.490+0000 7f107d547640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:21.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.490+0000 7f107d547640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:21.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.490+0000 7f107d547640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:37:21.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.490+0000 7f107d547640 1 -- 172.21.15.19:0/1759746974 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10780627b0 con 0x7f107814be90 2024-03-25T19:37:21.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.775+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1070002080 con 0x7f107814be90 2024-03-25T19:37:21.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.775+0000 7f105f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:21.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.775+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1070004d00 con 0x7f107814be90 2024-03-25T19:37:21.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.775+0000 7f105f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.775+0000 7f105f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:21.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.775+0000 7f105f7fe640 10 monclient: dump: 2024-03-25T19:37:21.775 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:21.776 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:21.776 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:21.776 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:21.776 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.776 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:21.776 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:21.776 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:21.776 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:21.776 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:21.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.775+0000 7f10757fa640 4 set_mon_vals no callback set 2024-03-25T19:37:21.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.776+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f1070004f20 con 0x7f107814be90 2024-03-25T19:37:21.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.776+0000 7f1076ffd640 10 monclient: get_auth_request con 0x7f106406de00 auth_method 0 2024-03-25T19:37:21.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.777+0000 7f105f7fe640 10 client.4594.objecter ms_handle_connect 0x7f106406de00 2024-03-25T19:37:21.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.785+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f1070054a10 con 0x7f107814be90 2024-03-25T19:37:21.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.785+0000 7f105f7fe640 10 client.4594.objecter ms_dispatch 0x7f1078000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:21.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.785+0000 7f105f7fe640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:21.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.785+0000 7f105f7fe640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:21.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.785+0000 7f105f7fe640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-25T19:37:21.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.786+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1070050350 con 0x7f107814be90 2024-03-25T19:37:21.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.786+0000 7f105f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:21.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:21.786+0000 7f105f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:22.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.058+0000 7f107d547640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:22.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.058+0000 7f107d547640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:22.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.058+0000 7f107d547640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:37:22.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.058+0000 7f107d547640 1 -- 172.21.15.19:0/1759746974 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10781db030 con 0x7f107814be90 2024-03-25T19:37:22.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.090+0000 7f105f7fe640 1 -- 172.21.15.19:0/1759746974 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f1070052450 con 0x7f107814be90 2024-03-25T19:37:22.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.090+0000 7f105f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:22.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.090+0000 7f105f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:22.090 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:37:22.090 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:22.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.092+0000 7f107d547640 10 librados: watch_flush enter 2024-03-25T19:37:22.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.093+0000 7f107d547640 10 librados: watch_flush exit 2024-03-25T19:37:22.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.093+0000 7f107d547640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-25T19:37:22.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.093+0000 7f107d547640 10 client.4594.objecter successfully canceled tick 2024-03-25T19:37:22.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.093+0000 7f107d547640 1 -- 172.21.15.19:0/1759746974 >> v1:172.21.15.188:6800/2207506229 conn(0x7f106406de00 legacy=0x7f10640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:22.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.093+0000 7f107d547640 10 monclient: shutdown 2024-03-25T19:37:22.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.093+0000 7f107d547640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:22.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.093+0000 7f107d547640 1 -- 172.21.15.19:0/1759746974 >> v1:172.21.15.188:6789/0 conn(0x7f107814be90 legacy=0x7f10781e2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:22.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.093+0000 7f107d547640 1 -- 172.21.15.19:0/1759746974 shutdown_connections 2024-03-25T19:37:22.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.093+0000 7f107d547640 1 -- 172.21.15.19:0/1759746974 >> 172.21.15.19:0/1759746974 conn(0x7f10780617a0 msgr2=0x7f10780b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:22.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.093+0000 7f107d547640 1 -- 172.21.15.19:0/1759746974 shutdown_connections 2024-03-25T19:37:22.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.093+0000 7f107d547640 1 -- 172.21.15.19:0/1759746974 wait complete. 2024-03-25T19:37:22.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:22.093+0000 7f107d547640 1 librados: shutdown 2024-03-25T19:37:22.104 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:28.911 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:29.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.024+0000 7fbada631640 10 monclient: get_monmap_and_config 2024-03-25T19:37:29.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.024+0000 7fbada631640 10 monclient: build_initial_monmap 2024-03-25T19:37:29.024 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.024+0000 7fbada631640 10 monclient: monmap: 2024-03-25T19:37:29.024 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:29.024 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:29.025 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:37:29.025731+0000 2024-03-25T19:37:29.025 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:37:29.025731+0000 2024-03-25T19:37:29.025 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:29.025 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:29.025 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:29.025 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:37:29.025 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:37:29.025 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:29.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.025+0000 7fbada631640 1 Processor -- start 2024-03-25T19:37:29.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.025+0000 7fbada631640 1 -- start start 2024-03-25T19:37:29.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.025+0000 7fbada631640 10 monclient: init 2024-03-25T19:37:29.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.026+0000 7fbada631640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:29.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.026+0000 7fbada631640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:29.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.026+0000 7fbada631640 10 monclient(hunting): picked mon.noname-a con 0x7fbad40b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:29.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.026+0000 7fbada631640 10 monclient(hunting): picked mon.noname-b con 0x7fbad40b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:29.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.026+0000 7fbada631640 10 monclient(hunting): picked mon.noname-c con 0x7fbad414be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:29.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.026+0000 7fbada631640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad40af940 con 0x7fbad40b4d60 2024-03-25T19:37:29.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.026+0000 7fbada631640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad40b0a70 con 0x7fbad414be90 2024-03-25T19:37:29.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.026+0000 7fbada631640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad40b0d50 con 0x7fbad40b1730 2024-03-25T19:37:29.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.026+0000 7fbada631640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:29.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.026+0000 7fbada631640 10 monclient(hunting): authenticate will time out at 6502.296131s 2024-03-25T19:37:29.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.034+0000 7fbad37fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fbad40b1730 0x7fbad40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:47856/0 (socket says 172.21.15.19:47856) 2024-03-25T19:37:29.034 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.034+0000 7fbad37fe640 1 -- 172.21.15.19:0/2489613813 learned_addr learned my addr 172.21.15.19:0/2489613813 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:29.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.056+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008861028 0 0) 0x7fbad40b0a70 con 0x7fbad414be90 2024-03-25T19:37:29.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.056+0000 7fbad2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:29.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.056+0000 7fbad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:29.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.056+0000 7fbad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:29.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.056+0000 7fbad2ffd640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:37:29.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.056+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbabc003520 con 0x7fbad414be90 2024-03-25T19:37:29.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.060+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466987138 0 0) 0x7fbad40b0d50 con 0x7fbad40b1730 2024-03-25T19:37:29.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.060+0000 7fbad2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:29.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.060+0000 7fbad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:29.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.060+0000 7fbad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:29.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.060+0000 7fbad2ffd640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:37:29.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.060+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad40b0a70 con 0x7fbad40b1730 2024-03-25T19:37:29.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.066+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621591713 0 0) 0x7fbad40af940 con 0x7fbad40b4d60 2024-03-25T19:37:29.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.067+0000 7fbad2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:29.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.067+0000 7fbad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:29.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.067+0000 7fbad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:29.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.067+0000 7fbad2ffd640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:37:29.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.067+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad40b0d50 con 0x7fbad40b4d60 2024-03-25T19:37:29.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3003413938 0 0) 0x7fbad40b0a70 con 0x7fbad40b1730 2024-03-25T19:37:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 >> v1:172.21.15.188:6789/0 conn(0x7fbad414be90 legacy=0x7fbad414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 >> v1:172.21.15.19:6789/0 conn(0x7fbad40b4d60 legacy=0x7fbad413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 1 monclient: found mon.noname-b 2024-03-25T19:37:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:37:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad40b1030 con 0x7fbad40b1730 2024-03-25T19:37:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fbac8001670 con 0x7fbad40b1730 2024-03-25T19:37:29.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbad2ffd640 10 monclient: dump: 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbada631640 5 monclient: authenticate success, global_id 4574 2024-03-25T19:37:29.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.069+0000 7fbada631640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:29.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.074+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbac80017d0 con 0x7fbad40b1730 2024-03-25T19:37:29.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.074+0000 7fbad2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.074+0000 7fbad2ffd640 1 -- 172.21.15.19:0/2489613813 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fbac8001be0 con 0x7fbad40b1730 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.074+0000 7fbad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.074+0000 7fbad2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.074+0000 7fbad2ffd640 10 monclient: dump: 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:29.075 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:29.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.074+0000 7fbada631640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:29.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.074+0000 7fbada631640 4 set_mon_vals no callback set 2024-03-25T19:37:29.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.075+0000 7fbada631640 10 monclient: shutdown 2024-03-25T19:37:29.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.075+0000 7fbada631640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:29.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.075+0000 7fbada631640 1 -- 172.21.15.19:0/2489613813 >> v1:172.21.15.19:6790/0 conn(0x7fbad40b1730 legacy=0x7fbad40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.075+0000 7fbada631640 1 -- 172.21.15.19:0/2489613813 shutdown_connections 2024-03-25T19:37:29.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.075+0000 7fbada631640 1 -- 172.21.15.19:0/2489613813 >> 172.21.15.19:0/2489613813 conn(0x7fbad40617a0 msgr2=0x7fbad4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:29.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.075+0000 7fbada631640 1 -- 172.21.15.19:0/2489613813 shutdown_connections 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.075+0000 7fbada631640 1 -- 172.21.15.19:0/2489613813 wait complete. 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.076+0000 7fbada631640 10 monclient: build_initial_monmap 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.076+0000 7fbada631640 10 monclient: monmap: 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.076+0000 7fbada631640 1 librados: starting msgr at 2024-03-25T19:37:29.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.076+0000 7fbada631640 1 librados: starting objecter 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.077+0000 7fbada631640 1 Processor -- start 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.077+0000 7fbada631640 1 -- start start 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.077+0000 7fbada631640 1 librados: setting wanted keys 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.077+0000 7fbada631640 1 librados: calling monclient init 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.077+0000 7fbada631640 10 monclient: init 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.077+0000 7fbada631640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.077+0000 7fbada631640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.077+0000 7fbada631640 10 monclient(hunting): picked mon.noname-b con 0x7fbad40b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.077+0000 7fbada631640 10 monclient(hunting): picked mon.noname-a con 0x7fbad40b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.077+0000 7fbada631640 10 monclient(hunting): picked mon.noname-c con 0x7fbad414be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.078+0000 7fbada631640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad41381d0 con 0x7fbad40b4d60 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.078+0000 7fbada631640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad41397e0 con 0x7fbad40b1730 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.078+0000 7fbada631640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad4137cd0 con 0x7fbad414be90 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.078+0000 7fbada631640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:29.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.078+0000 7fbada631640 10 monclient(hunting): authenticate will time out at 6502.347947s 2024-03-25T19:37:29.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.092+0000 7fbad3fff640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fbad40b1730 0x7fbad4135fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:37224/0 (socket says 172.21.15.19:37224) 2024-03-25T19:37:29.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.092+0000 7fbad3fff640 1 -- 172.21.15.19:0/2150515171 learned_addr learned my addr 172.21.15.19:0/2150515171 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:29.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.094+0000 7fbad0ff9640 10 client.?.objecter ms_handle_connect 0x7fbad40b1730 2024-03-25T19:37:29.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.094+0000 7fbad0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:29.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.094+0000 7fbad0ff9640 10 client.?.objecter ms_handle_connect 0x7fbad414be90 2024-03-25T19:37:29.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.094+0000 7fbad0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:29.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.100+0000 7fbad0ff9640 10 client.?.objecter ms_handle_connect 0x7fbad40b4d60 2024-03-25T19:37:29.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.100+0000 7fbad0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:29.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.101+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625693697 0 0) 0x7fbad4137cd0 con 0x7fbad414be90 2024-03-25T19:37:29.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.101+0000 7fbad0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:29.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.101+0000 7fbad0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:29.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.101+0000 7fbad0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:29.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.101+0000 7fbad0ff9640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:37:29.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.101+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac0004eb0 con 0x7fbad414be90 2024-03-25T19:37:29.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.102+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981744358 0 0) 0x7fbad41397e0 con 0x7fbad40b1730 2024-03-25T19:37:29.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.102+0000 7fbad0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:29.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.102+0000 7fbad0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:29.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.102+0000 7fbad0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:29.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.102+0000 7fbad0ff9640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:37:29.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.102+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad4137cd0 con 0x7fbad40b1730 2024-03-25T19:37:29.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.114+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565692110 0 0) 0x7fbad41381d0 con 0x7fbad40b4d60 2024-03-25T19:37:29.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.114+0000 7fbad0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:29.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.114+0000 7fbad0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:29.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.114+0000 7fbad0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:29.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.114+0000 7fbad0ff9640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:37:29.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.114+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad41397e0 con 0x7fbad40b4d60 2024-03-25T19:37:29.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.115+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2257018643 0 0) 0x7fbac0004eb0 con 0x7fbad414be90 2024-03-25T19:37:29.115 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.115+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbad41381d0 con 0x7fbad414be90 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.115+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fbacc001af0 con 0x7fbad414be90 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.115+0000 7fbad0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.115+0000 7fbad0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.115+0000 7fbad0ff9640 10 monclient(hunting): dump: 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:29.116 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:29.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.115+0000 7fbad0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:29.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.122+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2186657953 0 0) 0x7fbad4137cd0 con 0x7fbad40b1730 2024-03-25T19:37:29.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.122+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbac0004eb0 con 0x7fbad40b1730 2024-03-25T19:37:29.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.122+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fbac40019f0 con 0x7fbad40b1730 2024-03-25T19:37:29.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.122+0000 7fbad0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:29.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.122+0000 7fbad0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:29.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.122+0000 7fbad0ff9640 10 monclient(hunting): dump: 2024-03-25T19:37:29.122 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:29.122 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:29.122 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:29.122 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:29.122 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:29.123 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:29.123 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:29.123 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:29.123 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:29.123 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:29.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2642262985 0 0) 0x7fbad41381d0 con 0x7fbad414be90 2024-03-25T19:37:29.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 >> v1:172.21.15.188:6789/0 conn(0x7fbad40b1730 legacy=0x7fbad4135fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 >> v1:172.21.15.19:6789/0 conn(0x7fbad40b4d60 legacy=0x7fbad41366e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbad0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:37:29.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbad0ff9640 1 monclient: found mon.c 2024-03-25T19:37:29.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbad0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:29.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbad0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:29.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad4137e60 con 0x7fbad414be90 2024-03-25T19:37:29.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbad0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:37:29.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbad0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:37:29.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbad0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:29.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbada631640 5 monclient: authenticate success, global_id 4577 2024-03-25T19:37:29.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbada631640 10 monclient: _renew_subs 2024-03-25T19:37:29.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbada631640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:29.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbada631640 1 -- 172.21.15.19:0/2150515171 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad4138df0 con 0x7fbad414be90 2024-03-25T19:37:29.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbada631640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:29.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbada631640 10 monclient: _renew_subs 2024-03-25T19:37:29.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbada631640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:29.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbada631640 1 -- 172.21.15.19:0/2150515171 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad4145310 con 0x7fbad414be90 2024-03-25T19:37:29.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.124+0000 7fbada631640 1 librados: init done 2024-03-25T19:37:29.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.125+0000 7fbada631640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:29.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.125+0000 7fbada631640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:29.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.125+0000 7fbada631640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:29.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.125+0000 7fbada631640 1 -- 172.21.15.19:0/2150515171 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad41401f0 con 0x7fbad414be90 2024-03-25T19:37:29.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.455+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbacc001fe0 con 0x7fbad414be90 2024-03-25T19:37:29.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.455+0000 7fbad0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:29.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.455+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fbacc003b30 con 0x7fbad414be90 2024-03-25T19:37:29.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.455+0000 7fbad0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:29.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.455+0000 7fbad0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:29.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.455+0000 7fbad0ff9640 10 monclient: dump: 2024-03-25T19:37:29.456 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:29.456 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:29.456 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:29.456 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:29.456 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:29.456 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:29.456 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:29.456 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:29.456 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:29.456 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:29.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.455+0000 7fbad27fc640 4 set_mon_vals no callback set 2024-03-25T19:37:29.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.455+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fbacc019b00 con 0x7fbad414be90 2024-03-25T19:37:29.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.456+0000 7fbad3fff640 10 monclient: get_auth_request con 0x7fbac006df20 auth_method 0 2024-03-25T19:37:29.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.456+0000 7fbad0ff9640 10 client.4577.objecter ms_handle_connect 0x7fbac006df20 2024-03-25T19:37:29.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.462+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fbacc053710 con 0x7fbad414be90 2024-03-25T19:37:29.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.462+0000 7fbad0ff9640 10 client.4577.objecter ms_dispatch 0x7fbad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:29.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.462+0000 7fbad0ff9640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:29.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.462+0000 7fbad0ff9640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:29.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.462+0000 7fbad0ff9640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-25T19:37:29.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.462+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbacc04f220 con 0x7fbad414be90 2024-03-25T19:37:29.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.462+0000 7fbad0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:29.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.462+0000 7fbad0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:29.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.731+0000 7fbada631640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:29.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.731+0000 7fbada631640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:29.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.731+0000 7fbada631640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:29.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.731+0000 7fbada631640 1 -- 172.21.15.19:0/2150515171 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbad40627b0 con 0x7fbad414be90 2024-03-25T19:37:29.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.756+0000 7fbad0ff9640 1 -- 172.21.15.19:0/2150515171 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fbacc051150 con 0x7fbad414be90 2024-03-25T19:37:29.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.756+0000 7fbad0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:29.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.756+0000 7fbad0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:29.757 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:37:29.757 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:29.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.758+0000 7fbada631640 10 librados: watch_flush enter 2024-03-25T19:37:29.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.758+0000 7fbada631640 10 librados: watch_flush exit 2024-03-25T19:37:29.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.758+0000 7fbada631640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-25T19:37:29.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.758+0000 7fbada631640 10 client.4577.objecter successfully canceled tick 2024-03-25T19:37:29.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.759+0000 7fbada631640 1 -- 172.21.15.19:0/2150515171 >> v1:172.21.15.188:6800/2207506229 conn(0x7fbac006df20 legacy=0x7fbac00703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.759+0000 7fbada631640 10 monclient: shutdown 2024-03-25T19:37:29.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.759+0000 7fbada631640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:29.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.759+0000 7fbada631640 1 -- 172.21.15.19:0/2150515171 >> v1:172.21.15.19:6790/0 conn(0x7fbad414be90 legacy=0x7fbad41e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.759+0000 7fbada631640 1 -- 172.21.15.19:0/2150515171 shutdown_connections 2024-03-25T19:37:29.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.759+0000 7fbada631640 1 -- 172.21.15.19:0/2150515171 >> 172.21.15.19:0/2150515171 conn(0x7fbad40617a0 msgr2=0x7fbad40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:29.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.759+0000 7fbada631640 1 -- 172.21.15.19:0/2150515171 shutdown_connections 2024-03-25T19:37:29.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.759+0000 7fbada631640 1 -- 172.21.15.19:0/2150515171 wait complete. 2024-03-25T19:37:29.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:29.759+0000 7fbada631640 1 librados: shutdown 2024-03-25T19:37:29.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:36.578 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:36.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.694+0000 7f50d6da3640 10 monclient: get_monmap_and_config 2024-03-25T19:37:36.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.694+0000 7f50d6da3640 10 monclient: build_initial_monmap 2024-03-25T19:37:36.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.694+0000 7f50d6da3640 10 monclient: monmap: 2024-03-25T19:37:36.695 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:36.695 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:36.695 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:37:36.695795+0000 2024-03-25T19:37:36.695 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:37:36.695795+0000 2024-03-25T19:37:36.695 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:36.695 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:36.695 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:36.695 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:37:36.695 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:37:36.695 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:36.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.695+0000 7f50d6da3640 1 Processor -- start 2024-03-25T19:37:36.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.695+0000 7f50d6da3640 1 -- start start 2024-03-25T19:37:36.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.695+0000 7f50d6da3640 10 monclient: init 2024-03-25T19:37:36.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.696+0000 7f50d6da3640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:36.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.696+0000 7f50d6da3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:36.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.696+0000 7f50d6da3640 10 monclient(hunting): picked mon.noname-b con 0x7f50d00ae6e0 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:36.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.696+0000 7f50d6da3640 10 monclient(hunting): picked mon.noname-a con 0x7f50d013c900 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:36.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.697+0000 7f50d6da3640 10 monclient(hunting): picked mon.noname-c con 0x7f50d006f0b0 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:36.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.697+0000 7f50d6da3640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d006f7d0 con 0x7f50d013c900 2024-03-25T19:37:36.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.697+0000 7f50d6da3640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d00714d0 con 0x7f50d006f0b0 2024-03-25T19:37:36.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.697+0000 7f50d6da3640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d00717b0 con 0x7f50d00ae6e0 2024-03-25T19:37:36.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.697+0000 7f50d6da3640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:36.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.697+0000 7f50d6da3640 10 monclient(hunting): authenticate will time out at 6509.967083s 2024-03-25T19:37:36.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.705+0000 7f50cffff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f50d013c900 0x7f50d0071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:54500/0 (socket says 172.21.15.19:54500) 2024-03-25T19:37:36.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.705+0000 7f50cffff640 1 -- 172.21.15.19:0/1440923905 learned_addr learned my addr 172.21.15.19:0/1440923905 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:36.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.715+0000 7f50cf7fe640 1 -- 172.21.15.19:0/1440923905 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129152339 0 0) 0x7f50d00714d0 con 0x7f50d006f0b0 2024-03-25T19:37:36.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.715+0000 7f50cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.715+0000 7f50cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.715+0000 7f50cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.715+0000 7f50cf7fe640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:37:36.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.715+0000 7f50cf7fe640 1 -- 172.21.15.19:0/1440923905 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c0003520 con 0x7f50d006f0b0 2024-03-25T19:37:36.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.717+0000 7f50cf7fe640 1 -- 172.21.15.19:0/1440923905 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533290590 0 0) 0x7f50d006f7d0 con 0x7f50d013c900 2024-03-25T19:37:36.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.717+0000 7f50cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.717+0000 7f50cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.717+0000 7f50cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.717+0000 7f50cf7fe640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:37:36.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.717+0000 7f50cf7fe640 1 -- 172.21.15.19:0/1440923905 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d00714d0 con 0x7f50d013c900 2024-03-25T19:37:36.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 1 -- 172.21.15.19:0/1440923905 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3222190507 0 0) 0x7f50c0003520 con 0x7f50d006f0b0 2024-03-25T19:37:36.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 1 -- 172.21.15.19:0/1440923905 >> v1:172.21.15.19:6790/0 conn(0x7f50d00ae6e0 legacy=0x7f50d013c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 1 -- 172.21.15.19:0/1440923905 >> v1:172.21.15.19:6789/0 conn(0x7f50d013c900 legacy=0x7f50d0071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:36.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 1 monclient: found mon.noname-c 2024-03-25T19:37:36.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:36.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:37:36.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 1 -- 172.21.15.19:0/1440923905 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d0072470 con 0x7f50d006f0b0 2024-03-25T19:37:36.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:36.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:36.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:36.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 1 -- 172.21.15.19:0/1440923905 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f50c8001a10 con 0x7f50d006f0b0 2024-03-25T19:37:36.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50cf7fe640 10 monclient: dump: 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50d6da3640 5 monclient: authenticate success, global_id 4603 2024-03-25T19:37:36.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.727+0000 7f50d6da3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:36.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.743+0000 7f50cf7fe640 1 -- 172.21.15.19:0/1440923905 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50c8001b70 con 0x7f50d006f0b0 2024-03-25T19:37:36.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.743+0000 7f50cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:36.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.743+0000 7f50cf7fe640 1 -- 172.21.15.19:0/1440923905 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f50c8001f80 con 0x7f50d006f0b0 2024-03-25T19:37:36.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.743+0000 7f50d6da3640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:36.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.743+0000 7f50d6da3640 4 set_mon_vals no callback set 2024-03-25T19:37:36.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.743+0000 7f50cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.743+0000 7f50cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:36.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.743+0000 7f50cf7fe640 10 monclient: dump: 2024-03-25T19:37:36.743 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:36.744 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:36.744 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:36.744 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:36.744 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.744 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:36.744 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:36.744 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:36.744 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:36.744 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:36.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.744+0000 7f50d6da3640 10 monclient: shutdown 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.744+0000 7f50d6da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.744+0000 7f50d6da3640 1 -- 172.21.15.19:0/1440923905 >> v1:172.21.15.188:6789/0 conn(0x7f50d006f0b0 legacy=0x7f50d014a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.744+0000 7f50d6da3640 1 -- 172.21.15.19:0/1440923905 shutdown_connections 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.744+0000 7f50d6da3640 1 -- 172.21.15.19:0/1440923905 >> 172.21.15.19:0/1440923905 conn(0x7f50d006b630 msgr2=0x7f50d00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.744+0000 7f50d6da3640 1 -- 172.21.15.19:0/1440923905 shutdown_connections 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.744+0000 7f50d6da3640 1 -- 172.21.15.19:0/1440923905 wait complete. 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.745+0000 7f50d6da3640 10 monclient: build_initial_monmap 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.745+0000 7f50d6da3640 10 monclient: monmap: 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:36.746 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:37:36.747 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:36.747 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:37:36.747 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:37:36.747 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:36.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.745+0000 7f50d6da3640 1 librados: starting msgr at 2024-03-25T19:37:36.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.745+0000 7f50d6da3640 1 librados: starting objecter 2024-03-25T19:37:36.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.745+0000 7f50d6da3640 1 Processor -- start 2024-03-25T19:37:36.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.745+0000 7f50d6da3640 1 -- start start 2024-03-25T19:37:36.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.745+0000 7f50d6da3640 1 librados: setting wanted keys 2024-03-25T19:37:36.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.745+0000 7f50d6da3640 1 librados: calling monclient init 2024-03-25T19:37:36.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.745+0000 7f50d6da3640 10 monclient: init 2024-03-25T19:37:36.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.746+0000 7f50d6da3640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:36.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.746+0000 7f50d6da3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:36.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.746+0000 7f50d6da3640 10 monclient(hunting): picked mon.noname-c con 0x7f50d006f0b0 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:36.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.746+0000 7f50d6da3640 10 monclient(hunting): picked mon.noname-a con 0x7f50d00ae6e0 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:36.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.746+0000 7f50d6da3640 10 monclient(hunting): picked mon.noname-b con 0x7f50d013c900 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:36.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.746+0000 7f50d6da3640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d013d320 con 0x7f50d00ae6e0 2024-03-25T19:37:36.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.746+0000 7f50d6da3640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d00b3290 con 0x7f50d013c900 2024-03-25T19:37:36.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.746+0000 7f50d6da3640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d00b3570 con 0x7f50d006f0b0 2024-03-25T19:37:36.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.746+0000 7f50d6da3640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:36.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.746+0000 7f50d6da3640 10 monclient(hunting): authenticate will time out at 6510.016716s 2024-03-25T19:37:36.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.759+0000 7f50d4b3f640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f50d006f0b0 0x7f50d013d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:39246/0 (socket says 172.21.15.19:39246) 2024-03-25T19:37:36.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.759+0000 7f50d4b3f640 1 -- 172.21.15.19:0/2410487030 learned_addr learned my addr 172.21.15.19:0/2410487030 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:36.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.760+0000 7f50cd7fa640 10 client.?.objecter ms_handle_connect 0x7f50d006f0b0 2024-03-25T19:37:36.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.760+0000 7f50cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:36.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.763+0000 7f50cd7fa640 10 client.?.objecter ms_handle_connect 0x7f50d013c900 2024-03-25T19:37:36.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.763+0000 7f50cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:36.772 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.773+0000 7f50cd7fa640 10 client.?.objecter ms_handle_connect 0x7f50d00ae6e0 2024-03-25T19:37:36.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.773+0000 7f50cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:36.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.830+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366888215 0 0) 0x7f50d013d320 con 0x7f50d00ae6e0 2024-03-25T19:37:36.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.830+0000 7f50cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.830+0000 7f50cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.830+0000 7f50cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.830+0000 7f50cd7fa640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:37:36.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.830+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b4004eb0 con 0x7f50d00ae6e0 2024-03-25T19:37:36.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.836+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943912919 0 0) 0x7f50d00b3290 con 0x7f50d013c900 2024-03-25T19:37:36.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.836+0000 7f50cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.836+0000 7f50cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.836+0000 7f50cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.836+0000 7f50cd7fa640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:37:36.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.836+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d013d320 con 0x7f50d013c900 2024-03-25T19:37:36.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.854+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 194667552 0 0) 0x7f50d013d320 con 0x7f50d013c900 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.854+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50d00b3290 con 0x7f50d013c900 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.854+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f50c8001e80 con 0x7f50d013c900 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.854+0000 7f50cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.854+0000 7f50cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.854+0000 7f50cd7fa640 10 monclient(hunting): dump: 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:36.855 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:36.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.854+0000 7f50cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:36.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.869+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 433473410 0 0) 0x7f50d00b3290 con 0x7f50d013c900 2024-03-25T19:37:36.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.869+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 >> v1:172.21.15.19:6790/0 conn(0x7f50d006f0b0 legacy=0x7f50d013d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.869+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 >> v1:172.21.15.19:6789/0 conn(0x7f50d00ae6e0 legacy=0x7f50d012ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50cd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:37:36.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50cd7fa640 1 monclient: found mon.b 2024-03-25T19:37:36.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:36.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50cd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d0133a70 con 0x7f50d013c900 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50cd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50cd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50d6da3640 5 monclient: authenticate success, global_id 4606 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50d6da3640 10 monclient: _renew_subs 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50d6da3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50d6da3640 1 -- 172.21.15.19:0/2410487030 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d013ce40 con 0x7f50d013c900 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50d6da3640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50d6da3640 10 monclient: _renew_subs 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50d6da3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50d6da3640 1 -- 172.21.15.19:0/2410487030 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d0132910 con 0x7f50d013c900 2024-03-25T19:37:36.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.870+0000 7f50d6da3640 1 librados: init done 2024-03-25T19:37:36.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.871+0000 7f50d6da3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:36.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.871+0000 7f50d6da3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:36.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.871+0000 7f50d6da3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:37:36.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.871+0000 7f50d6da3640 1 -- 172.21.15.19:0/2410487030 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d013dec0 con 0x7f50d013c900 2024-03-25T19:37:36.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.891+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50c8001fe0 con 0x7f50d013c900 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.892+0000 7f50cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.892+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f50c8003c20 con 0x7f50d013c900 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.892+0000 7f50cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.892+0000 7f50cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.892+0000 7f50cd7fa640 10 monclient: dump: 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:36.892 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:36.893 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:36.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.892+0000 7f50ceffd640 4 set_mon_vals no callback set 2024-03-25T19:37:36.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.893+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f50c8019bf0 con 0x7f50d013c900 2024-03-25T19:37:36.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.893+0000 7f50d4b3f640 10 monclient: get_auth_request con 0x7f50b406dc90 auth_method 0 2024-03-25T19:37:36.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.893+0000 7f50cd7fa640 10 client.4606.objecter ms_handle_connect 0x7f50b406dc90 2024-03-25T19:37:36.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.894+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f50c8053830 con 0x7f50d013c900 2024-03-25T19:37:36.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.894+0000 7f50cd7fa640 10 client.4606.objecter ms_dispatch 0x7f50d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:36.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.894+0000 7f50cd7fa640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:36.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.894+0000 7f50cd7fa640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:36.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:36.895+0000 7f50cd7fa640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T19:37:37.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.157+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50c804f170 con 0x7f50d013c900 2024-03-25T19:37:37.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.157+0000 7f50cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:37.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.157+0000 7f50cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:37.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.428+0000 7f50d6da3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:37.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.428+0000 7f50d6da3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:37.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.428+0000 7f50d6da3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:37:37.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.428+0000 7f50d6da3640 1 -- 172.21.15.19:0/2410487030 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50d00627b0 con 0x7f50d013c900 2024-03-25T19:37:37.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.459+0000 7f50cd7fa640 1 -- 172.21.15.19:0/2410487030 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f50c8051270 con 0x7f50d013c900 2024-03-25T19:37:37.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.459+0000 7f50cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:37.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.459+0000 7f50cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:37.459 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:37:37.459 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:37.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.461+0000 7f50d6da3640 10 librados: watch_flush enter 2024-03-25T19:37:37.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.461+0000 7f50d6da3640 10 librados: watch_flush exit 2024-03-25T19:37:37.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.461+0000 7f50d6da3640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T19:37:37.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.461+0000 7f50d6da3640 10 client.4606.objecter successfully canceled tick 2024-03-25T19:37:37.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.461+0000 7f50d6da3640 1 -- 172.21.15.19:0/2410487030 >> v1:172.21.15.188:6800/2207506229 conn(0x7f50b406dc90 legacy=0x7f50b4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:37.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.461+0000 7f50d6da3640 10 monclient: shutdown 2024-03-25T19:37:37.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.461+0000 7f50d6da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:37.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.461+0000 7f50d6da3640 1 -- 172.21.15.19:0/2410487030 >> v1:172.21.15.188:6789/0 conn(0x7f50d013c900 legacy=0x7f50d01deb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:37.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.461+0000 7f50d6da3640 1 -- 172.21.15.19:0/2410487030 shutdown_connections 2024-03-25T19:37:37.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.462+0000 7f50d6da3640 1 -- 172.21.15.19:0/2410487030 >> 172.21.15.19:0/2410487030 conn(0x7f50d006b630 msgr2=0x7f50d0071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:37.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.462+0000 7f50d6da3640 1 -- 172.21.15.19:0/2410487030 shutdown_connections 2024-03-25T19:37:37.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.462+0000 7f50d6da3640 1 -- 172.21.15.19:0/2410487030 wait complete. 2024-03-25T19:37:37.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:37.462+0000 7f50d6da3640 1 librados: shutdown 2024-03-25T19:37:37.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:38.278 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:01.007 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.285 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:01.014 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.314 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:01.044 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.340 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:09:01.043 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.341 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:01.071 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.345 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:09:01.048 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.354 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:01.083 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.392 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:01.122 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.395 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:09:01.098 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:44.280 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:44.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.398+0000 7f004327b640 10 monclient: get_monmap_and_config 2024-03-25T19:37:44.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.398+0000 7f004327b640 10 monclient: build_initial_monmap 2024-03-25T19:37:44.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.398+0000 7f004327b640 10 monclient: monmap: 2024-03-25T19:37:44.398 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:44.398 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:44.398 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:37:44.399339+0000 2024-03-25T19:37:44.398 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:37:44.399339+0000 2024-03-25T19:37:44.398 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:44.398 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:44.398 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:44.398 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:37:44.398 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:37:44.399 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.398+0000 7f004327b640 1 Processor -- start 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.399+0000 7f004327b640 1 -- start start 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.399+0000 7f004327b640 10 monclient: init 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.399+0000 7f004327b640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.399+0000 7f004327b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.399+0000 7f004327b640 10 monclient(hunting): picked mon.noname-b con 0x7f003c0a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.399+0000 7f004327b640 10 monclient(hunting): picked mon.noname-a con 0x7f003c13f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.399+0000 7f004327b640 10 monclient(hunting): picked mon.noname-c con 0x7f003c1430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.399+0000 7f004327b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c142520 con 0x7f003c13f710 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.399+0000 7f004327b640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c0b28a0 con 0x7f003c1430c0 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.399+0000 7f004327b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c0b2a30 con 0x7f003c0a98f0 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.399+0000 7f004327b640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:44.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.399+0000 7f004327b640 10 monclient(hunting): authenticate will time out at 6517.669783s 2024-03-25T19:37:44.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.408+0000 7f0041818640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f003c1430c0 0x7f003c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:49580/0 (socket says 172.21.15.19:49580) 2024-03-25T19:37:44.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.408+0000 7f0041818640 1 -- 172.21.15.19:0/3881453497 learned_addr learned my addr 172.21.15.19:0/3881453497 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:44.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.421+0000 7f0033fff640 1 -- 172.21.15.19:0/3881453497 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678771139 0 0) 0x7f003c0b2a30 con 0x7f003c0a98f0 2024-03-25T19:37:44.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.422+0000 7f0033fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.422+0000 7f0033fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.422+0000 7f0033fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.422+0000 7f0033fff640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:37:44.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.422+0000 7f0033fff640 1 -- 172.21.15.19:0/3881453497 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0028003520 con 0x7f003c0a98f0 2024-03-25T19:37:44.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.425+0000 7f0033fff640 1 -- 172.21.15.19:0/3881453497 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575900599 0 0) 0x7f003c0b28a0 con 0x7f003c1430c0 2024-03-25T19:37:44.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.425+0000 7f0033fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.425+0000 7f0033fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.425+0000 7f0033fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.425+0000 7f0033fff640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:37:44.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.425+0000 7f0033fff640 1 -- 172.21.15.19:0/3881453497 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f003c0b2a30 con 0x7f003c1430c0 2024-03-25T19:37:44.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.436+0000 7f0033fff640 1 -- 172.21.15.19:0/3881453497 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548720786 0 0) 0x7f003c142520 con 0x7f003c13f710 2024-03-25T19:37:44.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.436+0000 7f0033fff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.436+0000 7f0033fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.436+0000 7f0033fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.436+0000 7f0033fff640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:37:44.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.436+0000 7f0033fff640 1 -- 172.21.15.19:0/3881453497 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f003c0b28a0 con 0x7f003c13f710 2024-03-25T19:37:44.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 1 -- 172.21.15.19:0/3881453497 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1105737341 0 0) 0x7f003c0b2a30 con 0x7f003c1430c0 2024-03-25T19:37:44.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 1 -- 172.21.15.19:0/3881453497 >> v1:172.21.15.19:6790/0 conn(0x7f003c0a98f0 legacy=0x7f003c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 1 -- 172.21.15.19:0/3881453497 >> v1:172.21.15.19:6789/0 conn(0x7f003c13f710 legacy=0x7f003c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 1 monclient: found mon.noname-c 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 1 -- 172.21.15.19:0/3881453497 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c0b44c0 con 0x7f003c1430c0 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 10 monclient: _finish_auth 0 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 1 -- 172.21.15.19:0/3881453497 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f00340019f0 con 0x7f003c1430c0 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f0033fff640 10 monclient: dump: 2024-03-25T19:37:44.438 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:44.439 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:44.439 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:44.439 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:44.439 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.439 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:44.439 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:44.439 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:44.439 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:44.439 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:44.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f004327b640 5 monclient: authenticate success, global_id 4609 2024-03-25T19:37:44.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.437+0000 7f004327b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:44.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.446+0000 7f0033fff640 1 -- 172.21.15.19:0/3881453497 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0034001b50 con 0x7f003c1430c0 2024-03-25T19:37:44.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.446+0000 7f0033fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:44.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.446+0000 7f004327b640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:44.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.446+0000 7f004327b640 4 set_mon_vals no callback set 2024-03-25T19:37:44.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.446+0000 7f004327b640 10 monclient: shutdown 2024-03-25T19:37:44.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.446+0000 7f004327b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:44.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.446+0000 7f004327b640 1 -- 172.21.15.19:0/3881453497 >> v1:172.21.15.188:6789/0 conn(0x7f003c1430c0 legacy=0x7f003c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.447+0000 7f004327b640 1 -- 172.21.15.19:0/3881453497 shutdown_connections 2024-03-25T19:37:44.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.447+0000 7f004327b640 1 -- 172.21.15.19:0/3881453497 >> 172.21.15.19:0/3881453497 conn(0x7f003c0b7ea0 msgr2=0x7f003c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:44.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.447+0000 7f004327b640 1 -- 172.21.15.19:0/3881453497 shutdown_connections 2024-03-25T19:37:44.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.447+0000 7f004327b640 1 -- 172.21.15.19:0/3881453497 wait complete. 2024-03-25T19:37:44.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.448+0000 7f004327b640 10 monclient: build_initial_monmap 2024-03-25T19:37:44.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.448+0000 7f004327b640 10 monclient: monmap: 2024-03-25T19:37:44.448 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:44.448 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:44.448 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:37:44.448 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:37:44.448 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:44.448 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:37:44.448 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:44.448 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:37:44.448 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:37:44.448 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:44.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.448+0000 7f004327b640 1 librados: starting msgr at 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.448+0000 7f004327b640 1 librados: starting objecter 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.448+0000 7f004327b640 1 Processor -- start 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.448+0000 7f004327b640 1 -- start start 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.448+0000 7f004327b640 1 librados: setting wanted keys 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.448+0000 7f004327b640 1 librados: calling monclient init 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.448+0000 7f004327b640 10 monclient: init 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.449+0000 7f004327b640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.449+0000 7f004327b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.449+0000 7f004327b640 10 monclient(hunting): picked mon.noname-c con 0x7f003c0a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.449+0000 7f004327b640 10 monclient(hunting): picked mon.noname-b con 0x7f003c13f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.449+0000 7f004327b640 10 monclient(hunting): picked mon.noname-a con 0x7f003c1430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.449+0000 7f004327b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c1349e0 con 0x7f003c1430c0 2024-03-25T19:37:44.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.449+0000 7f004327b640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c1389f0 con 0x7f003c13f710 2024-03-25T19:37:44.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.449+0000 7f004327b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c137bd0 con 0x7f003c0a98f0 2024-03-25T19:37:44.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.449+0000 7f004327b640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:44.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.449+0000 7f004327b640 10 monclient(hunting): authenticate will time out at 6517.719381s 2024-03-25T19:37:44.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.456+0000 7f0040816640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f003c13f710 0x7f003c133bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:49628/0 (socket says 172.21.15.19:49628) 2024-03-25T19:37:44.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.456+0000 7f0040816640 1 -- 172.21.15.19:0/4120822413 learned_addr learned my addr 172.21.15.19:0/4120822413 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:44.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.457+0000 7f0031ffb640 10 client.?.objecter ms_handle_connect 0x7f003c13f710 2024-03-25T19:37:44.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.457+0000 7f0031ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:44.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.466+0000 7f0031ffb640 10 client.?.objecter ms_handle_connect 0x7f003c0a98f0 2024-03-25T19:37:44.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.466+0000 7f0031ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:44.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.473+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816834912 0 0) 0x7f003c137bd0 con 0x7f003c0a98f0 2024-03-25T19:37:44.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.473+0000 7f0031ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.473+0000 7f0031ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.473+0000 7f0031ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.473+0000 7f0031ffb640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:37:44.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.473+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0020004eb0 con 0x7f003c0a98f0 2024-03-25T19:37:44.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.475+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092286793 0 0) 0x7f003c1389f0 con 0x7f003c13f710 2024-03-25T19:37:44.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.475+0000 7f0031ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.475+0000 7f0031ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.475+0000 7f0031ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.476 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.475+0000 7f0031ffb640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:37:44.476 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.475+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f003c137bd0 con 0x7f003c13f710 2024-03-25T19:37:44.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.483+0000 7f0031ffb640 10 client.?.objecter ms_handle_connect 0x7f003c1430c0 2024-03-25T19:37:44.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.483+0000 7f0031ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.488+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1991304410 0 0) 0x7f0020004eb0 con 0x7f003c0a98f0 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.488+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f003c1389f0 con 0x7f003c0a98f0 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.488+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0038001950 con 0x7f003c0a98f0 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.488+0000 7f0031ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.488+0000 7f0031ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.488+0000 7f0031ffb640 10 monclient(hunting): dump: 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:44.488 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:44.489 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:44.489 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:44.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.488+0000 7f0031ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:44.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.493+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054823054 0 0) 0x7f003c1349e0 con 0x7f003c1430c0 2024-03-25T19:37:44.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.493+0000 7f0031ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:44.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.493+0000 7f0031ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:44.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.493+0000 7f0031ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:44.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.493+0000 7f0031ffb640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:37:44.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.493+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0020004eb0 con 0x7f003c1430c0 2024-03-25T19:37:44.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3310149189 0 0) 0x7f003c1389f0 con 0x7f003c0a98f0 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 >> v1:172.21.15.188:6789/0 conn(0x7f003c13f710 legacy=0x7f003c133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 >> v1:172.21.15.19:6789/0 conn(0x7f003c1430c0 legacy=0x7f003c1342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f0031ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f0031ffb640 1 monclient: found mon.c 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f0031ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f0031ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c1db360 con 0x7f003c0a98f0 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f0031ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f0031ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f0031ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f004327b640 5 monclient: authenticate success, global_id 4589 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f004327b640 10 monclient: _renew_subs 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f004327b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f004327b640 1 -- 172.21.15.19:0/4120822413 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003c14e7a0 con 0x7f003c0a98f0 2024-03-25T19:37:44.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f004327b640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:44.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f004327b640 10 monclient: _renew_subs 2024-03-25T19:37:44.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f004327b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:44.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f004327b640 1 -- 172.21.15.19:0/4120822413 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f003c1390f0 con 0x7f003c0a98f0 2024-03-25T19:37:44.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.496+0000 7f004327b640 1 librados: init done 2024-03-25T19:37:44.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.497+0000 7f004327b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:44.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.497+0000 7f004327b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:44.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.497+0000 7f004327b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:44.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.497+0000 7f004327b640 1 -- 172.21.15.19:0/4120822413 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003c138b80 con 0x7f003c0a98f0 2024-03-25T19:37:44.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.858+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0038003610 con 0x7f003c0a98f0 2024-03-25T19:37:44.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.858+0000 7f0031ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:44.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.858+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0038001d80 con 0x7f003c0a98f0 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.858+0000 7f0031ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.858+0000 7f0031ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.858+0000 7f0031ffb640 10 monclient: dump: 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:44.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.858+0000 7f00337fe640 4 set_mon_vals no callback set 2024-03-25T19:37:44.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.859+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f0038003ae0 con 0x7f003c0a98f0 2024-03-25T19:37:44.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.860+0000 7f0040816640 10 monclient: get_auth_request con 0x7f002006de00 auth_method 0 2024-03-25T19:37:44.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.860+0000 7f0031ffb640 10 client.4589.objecter ms_handle_connect 0x7f002006de00 2024-03-25T19:37:44.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.865+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f0038053590 con 0x7f003c0a98f0 2024-03-25T19:37:44.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.865+0000 7f0031ffb640 10 client.4589.objecter ms_dispatch 0x7f003c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:44.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.865+0000 7f0031ffb640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:44.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.865+0000 7f0031ffb640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:44.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.865+0000 7f0031ffb640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T19:37:44.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.865+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f003804f0a0 con 0x7f003c0a98f0 2024-03-25T19:37:44.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.865+0000 7f0031ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:44.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:44.865+0000 7f0031ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:45.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.106+0000 7f004327b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:45.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.106+0000 7f004327b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:45.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.106+0000 7f004327b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:45.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.106+0000 7f004327b640 1 -- 172.21.15.19:0/4120822413 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f003c0627b0 con 0x7f003c0a98f0 2024-03-25T19:37:45.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.142+0000 7f0031ffb640 1 -- 172.21.15.19:0/4120822413 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f0038050fd0 con 0x7f003c0a98f0 2024-03-25T19:37:45.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.142+0000 7f0031ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:45.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.142+0000 7f0031ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:45.142 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:37:45.142 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:45.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 10 librados: watch_flush enter 2024-03-25T19:37:45.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 10 librados: watch_flush exit 2024-03-25T19:37:45.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T19:37:45.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 10 client.4589.objecter successfully canceled tick 2024-03-25T19:37:45.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 1 -- 172.21.15.19:0/4120822413 >> v1:172.21.15.188:6800/2207506229 conn(0x7f002006de00 legacy=0x7f00200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:45.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 10 monclient: shutdown 2024-03-25T19:37:45.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:45.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 1 -- 172.21.15.19:0/4120822413 >> v1:172.21.15.19:6790/0 conn(0x7f003c0a98f0 legacy=0x7f003c1dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:45.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 1 -- 172.21.15.19:0/4120822413 shutdown_connections 2024-03-25T19:37:45.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 1 -- 172.21.15.19:0/4120822413 >> 172.21.15.19:0/4120822413 conn(0x7f003c0b7ea0 msgr2=0x7f003c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:45.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 1 -- 172.21.15.19:0/4120822413 shutdown_connections 2024-03-25T19:37:45.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 1 -- 172.21.15.19:0/4120822413 wait complete. 2024-03-25T19:37:45.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:45.144+0000 7f004327b640 1 librados: shutdown 2024-03-25T19:37:45.157 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:51.963 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:52.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.079+0000 7f8cc7af1640 10 monclient: get_monmap_and_config 2024-03-25T19:37:52.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.079+0000 7f8cc7af1640 10 monclient: build_initial_monmap 2024-03-25T19:37:52.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.079+0000 7f8cc7af1640 10 monclient: monmap: 2024-03-25T19:37:52.079 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:52.080 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:52.080 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:37:52.080807+0000 2024-03-25T19:37:52.080 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:37:52.080807+0000 2024-03-25T19:37:52.080 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:52.080 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:52.080 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:52.080 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:37:52.080 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:37:52.080 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:52.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.080+0000 7f8cc7af1640 1 Processor -- start 2024-03-25T19:37:52.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.080+0000 7f8cc7af1640 1 -- start start 2024-03-25T19:37:52.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.080+0000 7f8cc7af1640 10 monclient: init 2024-03-25T19:37:52.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.081+0000 7f8cc7af1640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:52.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.081+0000 7f8cc7af1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:52.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.081+0000 7f8cc7af1640 10 monclient(hunting): picked mon.noname-a con 0x7f8cc00a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:52.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.081+0000 7f8cc7af1640 10 monclient(hunting): picked mon.noname-b con 0x7f8cc013f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:52.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.081+0000 7f8cc7af1640 10 monclient(hunting): picked mon.noname-c con 0x7f8cc01430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:52.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.081+0000 7f8cc7af1640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc0142520 con 0x7f8cc00a98f0 2024-03-25T19:37:52.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.081+0000 7f8cc7af1640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc00b28a0 con 0x7f8cc01430c0 2024-03-25T19:37:52.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.081+0000 7f8cc7af1640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc00b2a30 con 0x7f8cc013f710 2024-03-25T19:37:52.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.081+0000 7f8cc7af1640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:52.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.081+0000 7f8cc7af1640 10 monclient(hunting): authenticate will time out at 6525.351293s 2024-03-25T19:37:52.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.082+0000 7f8cc588d640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f8cc00a98f0 0x7f8cc00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:36114/0 (socket says 172.21.15.19:36114) 2024-03-25T19:37:52.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.082+0000 7f8cc588d640 1 -- 172.21.15.19:0/207472937 learned_addr learned my addr 172.21.15.19:0/207472937 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:52.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.092+0000 7f8cc488b640 1 -- 172.21.15.19:0/207472937 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215400087 0 0) 0x7f8cc0142520 con 0x7f8cc00a98f0 2024-03-25T19:37:52.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.092+0000 7f8cc488b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:52.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.092+0000 7f8cc488b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:52.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.092+0000 7f8cc488b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:52.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.092+0000 7f8cc488b640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:37:52.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.092+0000 7f8cc488b640 1 -- 172.21.15.19:0/207472937 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cac003520 con 0x7f8cc00a98f0 2024-03-25T19:37:52.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.098+0000 7f8cc488b640 1 -- 172.21.15.19:0/207472937 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284923257 0 0) 0x7f8cc00b28a0 con 0x7f8cc01430c0 2024-03-25T19:37:52.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.098+0000 7f8cc488b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:52.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.098+0000 7f8cc488b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:52.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.098+0000 7f8cc488b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:52.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.098+0000 7f8cc488b640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:37:52.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.098+0000 7f8cc488b640 1 -- 172.21.15.19:0/207472937 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc0142520 con 0x7f8cc01430c0 2024-03-25T19:37:52.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 1 -- 172.21.15.19:0/207472937 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2397936897 0 0) 0x7f8cac003520 con 0x7f8cc00a98f0 2024-03-25T19:37:52.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 1 -- 172.21.15.19:0/207472937 >> v1:172.21.15.19:6790/0 conn(0x7f8cc013f710 legacy=0x7f8cc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 1 -- 172.21.15.19:0/207472937 >> v1:172.21.15.188:6789/0 conn(0x7f8cc01430c0 legacy=0x7f8cc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 10 monclient: _finish_hunting 0 2024-03-25T19:37:52.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 1 monclient: found mon.noname-a 2024-03-25T19:37:52.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 1 -- 172.21.15.19:0/207472937 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc00b44c0 con 0x7f8cc00a98f0 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 10 monclient: _finish_auth 0 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 10 monclient: _check_auth_tickets 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 1 -- 172.21.15.19:0/207472937 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8cbc001690 con 0x7f8cc00a98f0 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc488b640 10 monclient: dump: 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:52.100 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:52.101 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:52.101 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:52.101 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:52.101 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:52.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc7af1640 5 monclient: authenticate success, global_id 4620 2024-03-25T19:37:52.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.099+0000 7f8cc7af1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:52.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.113+0000 7f8cc488b640 1 -- 172.21.15.19:0/207472937 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cbc0017f0 con 0x7f8cc00a98f0 2024-03-25T19:37:52.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.113+0000 7f8cc488b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:52.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.113+0000 7f8cc488b640 1 -- 172.21.15.19:0/207472937 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8cbc001c00 con 0x7f8cc00a98f0 2024-03-25T19:37:52.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.113+0000 7f8cc488b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.113+0000 7f8cc488b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.113+0000 7f8cc488b640 10 monclient: dump: 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.113+0000 7f8cc7af1640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:52.114 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.113+0000 7f8cc7af1640 4 set_mon_vals no callback set 2024-03-25T19:37:52.116 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.114+0000 7f8cc7af1640 10 monclient: shutdown 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.114+0000 7f8cc7af1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.114+0000 7f8cc7af1640 1 -- 172.21.15.19:0/207472937 >> v1:172.21.15.19:6789/0 conn(0x7f8cc00a98f0 legacy=0x7f8cc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.114+0000 7f8cc7af1640 1 -- 172.21.15.19:0/207472937 shutdown_connections 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.114+0000 7f8cc7af1640 1 -- 172.21.15.19:0/207472937 >> 172.21.15.19:0/207472937 conn(0x7f8cc00b7ea0 msgr2=0x7f8cc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.114+0000 7f8cc7af1640 1 -- 172.21.15.19:0/207472937 shutdown_connections 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.114+0000 7f8cc7af1640 1 -- 172.21.15.19:0/207472937 wait complete. 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.115+0000 7f8cc7af1640 10 monclient: build_initial_monmap 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.115+0000 7f8cc7af1640 10 monclient: monmap: 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:52.117 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.115+0000 7f8cc7af1640 1 librados: starting msgr at 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.115+0000 7f8cc7af1640 1 librados: starting objecter 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 1 Processor -- start 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 1 -- start start 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 1 librados: setting wanted keys 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 1 librados: calling monclient init 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 10 monclient: init 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 10 monclient(hunting): picked mon.noname-a con 0x7f8cc00a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:52.118 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 10 monclient(hunting): picked mon.noname-b con 0x7f8cc013f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:52.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 10 monclient(hunting): picked mon.noname-c con 0x7f8cc01430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:52.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc0132a30 con 0x7f8cc00a98f0 2024-03-25T19:37:52.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc0138c80 con 0x7f8cc013f710 2024-03-25T19:37:52.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc0138f60 con 0x7f8cc01430c0 2024-03-25T19:37:52.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:52.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.116+0000 7f8cc7af1640 10 monclient(hunting): authenticate will time out at 6525.386689s 2024-03-25T19:37:52.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.124+0000 7f8cc608e640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f8cc01430c0 0x7f8cc0132320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:44496/0 (socket says 172.21.15.19:44496) 2024-03-25T19:37:52.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.124+0000 7f8cc608e640 1 -- 172.21.15.19:0/1306071494 learned_addr learned my addr 172.21.15.19:0/1306071494 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:52.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.126+0000 7f8cb67fc640 10 client.?.objecter ms_handle_connect 0x7f8cc01430c0 2024-03-25T19:37:52.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.127+0000 7f8cb67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:52.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.132+0000 7f8cb67fc640 10 client.?.objecter ms_handle_connect 0x7f8cc013f710 2024-03-25T19:37:52.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.132+0000 7f8cb67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:52.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.140+0000 7f8cb67fc640 10 client.?.objecter ms_handle_connect 0x7f8cc00a98f0 2024-03-25T19:37:52.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.140+0000 7f8cb67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:52.140 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.140+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194388261 0 0) 0x7f8cc0138f60 con 0x7f8cc01430c0 2024-03-25T19:37:52.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.140+0000 7f8cb67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:52.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.140+0000 7f8cb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:52.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.140+0000 7f8cb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:52.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.140+0000 7f8cb67fc640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:37:52.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.140+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca4004eb0 con 0x7f8cc01430c0 2024-03-25T19:37:52.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.146+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959564854 0 0) 0x7f8cc0132a30 con 0x7f8cc00a98f0 2024-03-25T19:37:52.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.146+0000 7f8cb67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:52.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.146+0000 7f8cb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:52.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.146+0000 7f8cb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:52.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.146+0000 7f8cb67fc640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:37:52.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.146+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc0138f60 con 0x7f8cc00a98f0 2024-03-25T19:37:52.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.148+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023792008 0 0) 0x7f8cc0138c80 con 0x7f8cc013f710 2024-03-25T19:37:52.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.148+0000 7f8cb67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:52.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.148+0000 7f8cb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:52.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.148+0000 7f8cb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:52.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.148+0000 7f8cb67fc640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:37:52.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.148+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc0132a30 con 0x7f8cc013f710 2024-03-25T19:37:52.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.153+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1779429003 0 0) 0x7f8cc0138f60 con 0x7f8cc00a98f0 2024-03-25T19:37:52.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.153+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cc0138c80 con 0x7f8cc00a98f0 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.153+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8cbc001b00 con 0x7f8cc00a98f0 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.153+0000 7f8cb67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.153+0000 7f8cb67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.153+0000 7f8cb67fc640 10 monclient(hunting): dump: 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:52.154 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:52.155 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:52.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.153+0000 7f8cb67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:52.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.160+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2850396900 0 0) 0x7f8ca4004eb0 con 0x7f8cc01430c0 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.160+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cc0138f60 con 0x7f8cc01430c0 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.160+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8cb8001c90 con 0x7f8cc01430c0 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.160+0000 7f8cb67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.160+0000 7f8cb67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.160+0000 7f8cb67fc640 10 monclient(hunting): dump: 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:52.161 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:52.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.167+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 599477595 0 0) 0x7f8cc0132a30 con 0x7f8cc013f710 2024-03-25T19:37:52.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.167+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ca4004eb0 con 0x7f8cc013f710 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.167+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8cb0001830 con 0x7f8cc013f710 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.167+0000 7f8cb67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.167+0000 7f8cb67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.167+0000 7f8cb67fc640 10 monclient(hunting): dump: 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:52.168 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:52.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1976589800 0 0) 0x7f8cc0138c80 con 0x7f8cc00a98f0 2024-03-25T19:37:52.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 >> v1:172.21.15.19:6790/0 conn(0x7f8cc01430c0 legacy=0x7f8cc0132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 >> v1:172.21.15.188:6789/0 conn(0x7f8cc013f710 legacy=0x7f8cc0131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cb67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:52.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cb67fc640 1 monclient: found mon.a 2024-03-25T19:37:52.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:52.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cb67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:37:52.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc0139240 con 0x7f8cc00a98f0 2024-03-25T19:37:52.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cb67fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:52.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cb67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:52.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:52.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cc7af1640 5 monclient: authenticate success, global_id 4623 2024-03-25T19:37:52.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cc7af1640 10 monclient: _renew_subs 2024-03-25T19:37:52.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cc7af1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:37:52.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cc7af1640 1 -- 172.21.15.19:0/1306071494 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc0136e50 con 0x7f8cc00a98f0 2024-03-25T19:37:52.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cc7af1640 10 client.4623.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:52.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cc7af1640 10 monclient: _renew_subs 2024-03-25T19:37:52.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cc7af1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:37:52.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.171+0000 7f8cc7af1640 1 -- 172.21.15.19:0/1306071494 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc01389f0 con 0x7f8cc00a98f0 2024-03-25T19:37:52.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.172+0000 7f8cc7af1640 1 librados: init done 2024-03-25T19:37:52.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.173+0000 7f8cc7af1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:52.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.173+0000 7f8cc7af1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:52.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.173+0000 7f8cc7af1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:37:52.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.173+0000 7f8cc7af1640 1 -- 172.21.15.19:0/1306071494 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc00627b0 con 0x7f8cc00a98f0 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.492+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cbc004a90 con 0x7f8cc00a98f0 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.492+0000 7f8cb67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.492+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f8cbc001cc0 con 0x7f8cc00a98f0 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.492+0000 7f8cb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.492+0000 7f8cb67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.492+0000 7f8cb67fc640 10 monclient: dump: 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:52.493 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:52.494 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:52.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.492+0000 7f8cb7fff640 4 set_mon_vals no callback set 2024-03-25T19:37:52.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.493+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f8cbc01ac10 con 0x7f8cc00a98f0 2024-03-25T19:37:52.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.494+0000 7f8cc508c640 10 monclient: get_auth_request con 0x7f8ca406e0b0 auth_method 0 2024-03-25T19:37:52.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.494+0000 7f8cb67fc640 10 client.4623.objecter ms_handle_connect 0x7f8ca406e0b0 2024-03-25T19:37:52.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.500+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f8cbc0547f0 con 0x7f8cc00a98f0 2024-03-25T19:37:52.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.500+0000 7f8cb67fc640 10 client.4623.objecter ms_dispatch 0x7f8cc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:52.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.500+0000 7f8cb67fc640 3 client.4623.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:52.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.500+0000 7f8cb67fc640 3 client.4623.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:52.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.500+0000 7f8cb67fc640 20 client.4623.objecter dump_active .. 0 homeless 2024-03-25T19:37:52.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.500+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8cbc050130 con 0x7f8cc00a98f0 2024-03-25T19:37:52.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.500+0000 7f8cb67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:52.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.500+0000 7f8cb67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:52.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.750+0000 7f8cc7af1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:52.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.751+0000 7f8cc7af1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:52.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.751+0000 7f8cc7af1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:37:52.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.751+0000 7f8cc7af1640 1 -- 172.21.15.19:0/1306071494 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cc013dd60 con 0x7f8cc00a98f0 2024-03-25T19:37:52.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.783+0000 7f8cb67fc640 1 -- 172.21.15.19:0/1306071494 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f8cbc052230 con 0x7f8cc00a98f0 2024-03-25T19:37:52.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.783+0000 7f8cb67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:52.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.783+0000 7f8cb67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:52.783 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:37:52.784 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:52.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 10 librados: watch_flush enter 2024-03-25T19:37:52.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 10 librados: watch_flush exit 2024-03-25T19:37:52.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 20 client.4623.objecter shutdown clearing up homeless session... 2024-03-25T19:37:52.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 10 client.4623.objecter successfully canceled tick 2024-03-25T19:37:52.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 1 -- 172.21.15.19:0/1306071494 >> v1:172.21.15.188:6800/2207506229 conn(0x7f8ca406e0b0 legacy=0x7f8ca4070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 10 monclient: shutdown 2024-03-25T19:37:52.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:52.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 1 -- 172.21.15.19:0/1306071494 >> v1:172.21.15.19:6789/0 conn(0x7f8cc00a98f0 legacy=0x7f8cc0137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 1 -- 172.21.15.19:0/1306071494 shutdown_connections 2024-03-25T19:37:52.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 1 -- 172.21.15.19:0/1306071494 >> 172.21.15.19:0/1306071494 conn(0x7f8cc00b7ea0 msgr2=0x7f8cc0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:52.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 1 -- 172.21.15.19:0/1306071494 shutdown_connections 2024-03-25T19:37:52.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 1 -- 172.21.15.19:0/1306071494 wait complete. 2024-03-25T19:37:52.787 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:52.786+0000 7f8cc7af1640 1 librados: shutdown 2024-03-25T19:37:52.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:58.300 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:21.030 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.351 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:21.080 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.385 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:21.114 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.401 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:09:21.104 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.410 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:09:21.113 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.440 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:21.169 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.448 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:09:21.151 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.449 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:21.179 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.478 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:21.208 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:59.606 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:59.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.730+0000 7faa0d57c640 10 monclient: get_monmap_and_config 2024-03-25T19:37:59.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.730+0000 7faa0d57c640 10 monclient: build_initial_monmap 2024-03-25T19:37:59.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.730+0000 7faa0d57c640 10 monclient: monmap: 2024-03-25T19:37:59.731 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:59.731 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:59.731 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:37:59.731920+0000 2024-03-25T19:37:59.731 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:37:59.731920+0000 2024-03-25T19:37:59.731 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:59.731 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:59.731 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:59.731 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:37:59.731 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:37:59.731 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:59.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.731+0000 7faa0d57c640 1 Processor -- start 2024-03-25T19:37:59.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.731+0000 7faa0d57c640 1 -- start start 2024-03-25T19:37:59.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.731+0000 7faa0d57c640 10 monclient: init 2024-03-25T19:37:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.732+0000 7faa0d57c640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.732+0000 7faa0d57c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.732+0000 7faa0d57c640 10 monclient(hunting): picked mon.noname-c con 0x7faa080a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.732+0000 7faa0d57c640 10 monclient(hunting): picked mon.noname-a con 0x7faa0813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.732+0000 7faa0d57c640 10 monclient(hunting): picked mon.noname-b con 0x7faa081430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.732+0000 7faa0d57c640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa08142520 con 0x7faa0813f710 2024-03-25T19:37:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.732+0000 7faa0d57c640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa080b28a0 con 0x7faa080a98f0 2024-03-25T19:37:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.732+0000 7faa0d57c640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa080b2a30 con 0x7faa081430c0 2024-03-25T19:37:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.732+0000 7faa0d57c640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.732+0000 7faa0d57c640 10 monclient(hunting): authenticate will time out at 6533.002604s 2024-03-25T19:37:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.739+0000 7faa06ffd640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7faa080a98f0 0x7faa080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:54044/0 (socket says 172.21.15.19:54044) 2024-03-25T19:37:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.739+0000 7faa06ffd640 1 -- 172.21.15.19:0/3466385126 learned_addr learned my addr 172.21.15.19:0/3466385126 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:59.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.755+0000 7faa05ffb640 1 -- 172.21.15.19:0/3466385126 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279686858 0 0) 0x7faa080b28a0 con 0x7faa080a98f0 2024-03-25T19:37:59.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.755+0000 7faa05ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.755+0000 7faa05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.755+0000 7faa05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.755+0000 7faa05ffb640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:37:59.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.755+0000 7faa05ffb640 1 -- 172.21.15.19:0/3466385126 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9f0003520 con 0x7faa080a98f0 2024-03-25T19:37:59.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.766+0000 7faa05ffb640 1 -- 172.21.15.19:0/3466385126 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546664978 0 0) 0x7faa080b2a30 con 0x7faa081430c0 2024-03-25T19:37:59.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.766+0000 7faa05ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.766+0000 7faa05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.766+0000 7faa05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.766+0000 7faa05ffb640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:37:59.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.767+0000 7faa05ffb640 1 -- 172.21.15.19:0/3466385126 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa080b28a0 con 0x7faa081430c0 2024-03-25T19:37:59.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 1 -- 172.21.15.19:0/3466385126 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1166431930 0 0) 0x7fa9f0003520 con 0x7faa080a98f0 2024-03-25T19:37:59.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 1 -- 172.21.15.19:0/3466385126 >> v1:172.21.15.19:6790/0 conn(0x7faa081430c0 legacy=0x7faa0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 1 -- 172.21.15.19:0/3466385126 >> v1:172.21.15.19:6789/0 conn(0x7faa0813f710 legacy=0x7faa081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 1 monclient: found mon.noname-c 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 1 -- 172.21.15.19:0/3466385126 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa080b44c0 con 0x7faa080a98f0 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 1 -- 172.21.15.19:0/3466385126 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa9f8001690 con 0x7faa080a98f0 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa05ffb640 10 monclient: dump: 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:59.775 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:59.775 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:59.775 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:59.775 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.775 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:59.775 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:59.775 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:59.775 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:59.775 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:59.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa0d57c640 5 monclient: authenticate success, global_id 4621 2024-03-25T19:37:59.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.773+0000 7faa0d57c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:59.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.811+0000 7faa05ffb640 1 -- 172.21.15.19:0/3466385126 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9f80017f0 con 0x7faa080a98f0 2024-03-25T19:37:59.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.811+0000 7faa05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:59.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.811+0000 7faa05ffb640 1 -- 172.21.15.19:0/3466385126 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa9f8001c00 con 0x7faa080a98f0 2024-03-25T19:37:59.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.811+0000 7faa05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.811+0000 7faa05ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:59.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.811+0000 7faa05ffb640 10 monclient: dump: 2024-03-25T19:37:59.812 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:59.812 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:59.812 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:59.814 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:59.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.811+0000 7faa0d57c640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:59.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.811+0000 7faa0d57c640 4 set_mon_vals no callback set 2024-03-25T19:37:59.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.812+0000 7faa0d57c640 10 monclient: shutdown 2024-03-25T19:37:59.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.812+0000 7faa0d57c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:59.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.812+0000 7faa0d57c640 1 -- 172.21.15.19:0/3466385126 >> v1:172.21.15.188:6789/0 conn(0x7faa080a98f0 legacy=0x7faa080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.812+0000 7faa0d57c640 1 -- 172.21.15.19:0/3466385126 shutdown_connections 2024-03-25T19:37:59.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.812+0000 7faa0d57c640 1 -- 172.21.15.19:0/3466385126 >> 172.21.15.19:0/3466385126 conn(0x7faa080b7ea0 msgr2=0x7faa080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:59.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.812+0000 7faa0d57c640 1 -- 172.21.15.19:0/3466385126 shutdown_connections 2024-03-25T19:37:59.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.812+0000 7faa0d57c640 1 -- 172.21.15.19:0/3466385126 wait complete. 2024-03-25T19:37:59.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.813+0000 7faa0d57c640 10 monclient: build_initial_monmap 2024-03-25T19:37:59.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.813+0000 7faa0d57c640 10 monclient: monmap: 2024-03-25T19:37:59.820 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:37:59.820 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:59.821 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:37:59.821 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:37:59.821 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:59.821 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:37:59.821 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:37:59.821 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:37:59.822 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:37:59.822 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:59.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.813+0000 7faa0d57c640 1 librados: starting msgr at 2024-03-25T19:37:59.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.813+0000 7faa0d57c640 1 librados: starting objecter 2024-03-25T19:37:59.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.813+0000 7faa0d57c640 1 Processor -- start 2024-03-25T19:37:59.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.813+0000 7faa0d57c640 1 -- start start 2024-03-25T19:37:59.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.813+0000 7faa0d57c640 1 librados: setting wanted keys 2024-03-25T19:37:59.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.813+0000 7faa0d57c640 1 librados: calling monclient init 2024-03-25T19:37:59.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.813+0000 7faa0d57c640 10 monclient: init 2024-03-25T19:37:59.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.814+0000 7faa0d57c640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:59.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.814+0000 7faa0d57c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:59.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.814+0000 7faa0d57c640 10 monclient(hunting): picked mon.noname-a con 0x7faa080a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:37:59.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.814+0000 7faa0d57c640 10 monclient(hunting): picked mon.noname-c con 0x7faa0813f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:37:59.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.814+0000 7faa0d57c640 10 monclient(hunting): picked mon.noname-b con 0x7faa081430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:37:59.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.814+0000 7faa0d57c640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa08139a70 con 0x7faa080a98f0 2024-03-25T19:37:59.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.814+0000 7faa0d57c640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa08136b50 con 0x7faa081430c0 2024-03-25T19:37:59.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.814+0000 7faa0d57c640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa08136e30 con 0x7faa0813f710 2024-03-25T19:37:59.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.814+0000 7faa0d57c640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:59.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.814+0000 7faa0d57c640 10 monclient(hunting): authenticate will time out at 6533.084677s 2024-03-25T19:37:59.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.823+0000 7faa077fe640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7faa081430c0 0x7faa081e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:54092/0 (socket says 172.21.15.19:54092) 2024-03-25T19:37:59.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.823+0000 7faa077fe640 1 -- 172.21.15.19:0/4114658096 learned_addr learned my addr 172.21.15.19:0/4114658096 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:37:59.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.825+0000 7fa9ef7fe640 10 client.?.objecter ms_handle_connect 0x7faa0813f710 2024-03-25T19:37:59.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.825+0000 7fa9ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:59.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.826+0000 7fa9ef7fe640 10 client.?.objecter ms_handle_connect 0x7faa081430c0 2024-03-25T19:37:59.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.826+0000 7fa9ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:59.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.838+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573866084 0 0) 0x7faa08136b50 con 0x7faa081430c0 2024-03-25T19:37:59.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.838+0000 7fa9ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.838+0000 7fa9ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.838+0000 7fa9ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.838+0000 7fa9ef7fe640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:37:59.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.838+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9f4004eb0 con 0x7faa081430c0 2024-03-25T19:37:59.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.838+0000 7fa9ef7fe640 10 client.?.objecter ms_handle_connect 0x7faa080a98f0 2024-03-25T19:37:59.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.838+0000 7fa9ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:59.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.842+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641261725 0 0) 0x7faa08136e30 con 0x7faa0813f710 2024-03-25T19:37:59.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.842+0000 7fa9ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.842+0000 7fa9ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.842+0000 7fa9ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.842+0000 7fa9ef7fe640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:37:59.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.842+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa08136b50 con 0x7faa0813f710 2024-03-25T19:37:59.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.851+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712258435 0 0) 0x7faa08139a70 con 0x7faa080a98f0 2024-03-25T19:37:59.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.851+0000 7fa9ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.851+0000 7fa9ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.851+0000 7fa9ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.851+0000 7fa9ef7fe640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:37:59.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.851+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa08136e30 con 0x7faa080a98f0 2024-03-25T19:37:59.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.858+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1726797903 0 0) 0x7faa08136b50 con 0x7faa0813f710 2024-03-25T19:37:59.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.858+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa08139a70 con 0x7faa0813f710 2024-03-25T19:37:59.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.858+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa9fc0018a0 con 0x7faa0813f710 2024-03-25T19:37:59.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.858+0000 7fa9ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.858+0000 7fa9ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:59.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.858+0000 7fa9ef7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:59.859 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:59.859 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:59.859 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:59.859 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:59.860 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.860 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:59.860 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:59.860 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:59.860 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:59.860 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:59.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.858+0000 7fa9ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:59.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.861+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2311447531 0 0) 0x7fa9f4004eb0 con 0x7faa081430c0 2024-03-25T19:37:59.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.861+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa08136b50 con 0x7faa081430c0 2024-03-25T19:37:59.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.861+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7faa00001c90 con 0x7faa081430c0 2024-03-25T19:37:59.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.861+0000 7fa9ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.861+0000 7fa9ef7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:59.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.861+0000 7fa9ef7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:59.862 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:37:59.863 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:37:59.863 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:59.863 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:37:59.863 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.863 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:37:59.863 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:37:59.864 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:37:59.864 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:37:59.864 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:37:59.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 760138132 0 0) 0x7faa08139a70 con 0x7faa0813f710 2024-03-25T19:37:59.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 >> v1:172.21.15.188:6789/0 conn(0x7faa081430c0 legacy=0x7faa081e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 >> v1:172.21.15.19:6789/0 conn(0x7faa080a98f0 legacy=0x7faa08137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7fa9ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:59.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7fa9ef7fe640 1 monclient: found mon.c 2024-03-25T19:37:59.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7fa9ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:59.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7fa9ef7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:59.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa08137110 con 0x7faa0813f710 2024-03-25T19:37:59.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7fa9ef7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:59.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7fa9ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:59.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7fa9ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:59.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7faa0d57c640 5 monclient: authenticate success, global_id 4601 2024-03-25T19:37:59.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7faa0d57c640 10 monclient: _renew_subs 2024-03-25T19:37:59.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7faa0d57c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:59.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7faa0d57c640 1 -- 172.21.15.19:0/4114658096 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa080b7890 con 0x7faa0813f710 2024-03-25T19:37:59.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7faa0d57c640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:59.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7faa0d57c640 10 monclient: _renew_subs 2024-03-25T19:37:59.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7faa0d57c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:59.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7faa0d57c640 1 -- 172.21.15.19:0/4114658096 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa08135f60 con 0x7faa0813f710 2024-03-25T19:37:59.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.873+0000 7faa0d57c640 1 librados: init done 2024-03-25T19:37:59.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.874+0000 7faa0d57c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:59.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.874+0000 7faa0d57c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:59.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.874+0000 7faa0d57c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:37:59.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:37:59.874+0000 7faa0d57c640 1 -- 172.21.15.19:0/4114658096 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0813d010 con 0x7faa0813f710 2024-03-25T19:38:00.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.205+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9fc001d50 con 0x7faa0813f710 2024-03-25T19:38:00.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.205+0000 7fa9ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:00.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.205+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fa9fc0037d0 con 0x7faa0813f710 2024-03-25T19:38:00.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.205+0000 7fa9ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:00.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.205+0000 7fa9ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:00.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.205+0000 7fa9ef7fe640 10 monclient: dump: 2024-03-25T19:38:00.206 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:00.206 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:00.206 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:00.207 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:00.207 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:00.207 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:00.207 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:00.207 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:00.207 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:00.207 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:00.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.205+0000 7faa057fa640 4 set_mon_vals no callback set 2024-03-25T19:38:00.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.206+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fa9fc0039f0 con 0x7faa0813f710 2024-03-25T19:38:00.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.207+0000 7faa06ffd640 10 monclient: get_auth_request con 0x7fa9f406df20 auth_method 0 2024-03-25T19:38:00.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.207+0000 7fa9ef7fe640 10 client.4601.objecter ms_handle_connect 0x7fa9f406df20 2024-03-25T19:38:00.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.212+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fa9fc053550 con 0x7faa0813f710 2024-03-25T19:38:00.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.212+0000 7fa9ef7fe640 10 client.4601.objecter ms_dispatch 0x7faa08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:00.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.212+0000 7fa9ef7fe640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:00.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.212+0000 7fa9ef7fe640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:00.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.212+0000 7fa9ef7fe640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-25T19:38:00.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.212+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9fc04ee90 con 0x7faa0813f710 2024-03-25T19:38:00.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.212+0000 7fa9ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:00.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.212+0000 7fa9ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:00.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.467+0000 7faa0d57c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:00.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.467+0000 7faa0d57c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:00.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.467+0000 7faa0d57c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:38:00.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.467+0000 7faa0d57c640 1 -- 172.21.15.19:0/4114658096 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa080b7ad0 con 0x7faa0813f710 2024-03-25T19:38:00.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.523+0000 7fa9ef7fe640 1 -- 172.21.15.19:0/4114658096 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7faa080b7ad0 con 0x7faa0813f710 2024-03-25T19:38:00.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.523+0000 7fa9ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:00.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.523+0000 7fa9ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:00.524 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:38:00.524 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 10 librados: watch_flush enter 2024-03-25T19:38:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 10 librados: watch_flush exit 2024-03-25T19:38:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-25T19:38:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 10 client.4601.objecter successfully canceled tick 2024-03-25T19:38:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 1 -- 172.21.15.19:0/4114658096 >> v1:172.21.15.188:6800/2207506229 conn(0x7fa9f406df20 legacy=0x7fa9f40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 10 monclient: shutdown 2024-03-25T19:38:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 1 -- 172.21.15.19:0/4114658096 >> v1:172.21.15.19:6790/0 conn(0x7faa0813f710 legacy=0x7faa08135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 1 -- 172.21.15.19:0/4114658096 shutdown_connections 2024-03-25T19:38:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 1 -- 172.21.15.19:0/4114658096 >> 172.21.15.19:0/4114658096 conn(0x7faa080b7ea0 msgr2=0x7faa08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 1 -- 172.21.15.19:0/4114658096 shutdown_connections 2024-03-25T19:38:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 1 -- 172.21.15.19:0/4114658096 wait complete. 2024-03-25T19:38:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:00.526+0000 7faa0d57c640 1 librados: shutdown 2024-03-25T19:38:00.538 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:07.346 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:07.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.461+0000 7f49f0e81640 10 monclient: get_monmap_and_config 2024-03-25T19:38:07.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.461+0000 7f49f0e81640 10 monclient: build_initial_monmap 2024-03-25T19:38:07.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.461+0000 7f49f0e81640 10 monclient: monmap: 2024-03-25T19:38:07.462 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:07.462 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:07.462 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:38:07.462769+0000 2024-03-25T19:38:07.462 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:38:07.462769+0000 2024-03-25T19:38:07.462 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:07.462 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:07.462 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:07.462 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:38:07.462 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:38:07.462 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:07.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.462+0000 7f49f0e81640 1 Processor -- start 2024-03-25T19:38:07.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.462+0000 7f49f0e81640 1 -- start start 2024-03-25T19:38:07.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.462+0000 7f49f0e81640 10 monclient: init 2024-03-25T19:38:07.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.463+0000 7f49f0e81640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:07.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.463+0000 7f49f0e81640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:07.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.463+0000 7f49f0e81640 10 monclient(hunting): picked mon.noname-a con 0x7f49ec0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:07.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.463+0000 7f49f0e81640 10 monclient(hunting): picked mon.noname-b con 0x7f49ec13f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:07.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.463+0000 7f49f0e81640 10 monclient(hunting): picked mon.noname-c con 0x7f49ec1430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:07.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.463+0000 7f49f0e81640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ec142520 con 0x7f49ec0a98f0 2024-03-25T19:38:07.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.464+0000 7f49f0e81640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ec0b28a0 con 0x7f49ec1430c0 2024-03-25T19:38:07.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.464+0000 7f49f0e81640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ec0b2a30 con 0x7f49ec13f710 2024-03-25T19:38:07.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.464+0000 7f49f0e81640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:07.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.464+0000 7f49f0e81640 10 monclient(hunting): authenticate will time out at 6540.733975s 2024-03-25T19:38:07.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.482+0000 7f49e9d9b640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f49ec13f710 0x7f49ec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:46644/0 (socket says 172.21.15.19:46644) 2024-03-25T19:38:07.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.482+0000 7f49e9d9b640 1 -- 172.21.15.19:0/2221390617 learned_addr learned my addr 172.21.15.19:0/2221390617 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:07.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.493+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012452359 0 0) 0x7f49ec142520 con 0x7f49ec0a98f0 2024-03-25T19:38:07.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.493+0000 7f49e959a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.493+0000 7f49e959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.493+0000 7f49e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.493+0000 7f49e959a640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:38:07.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.493+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49d4003520 con 0x7f49ec0a98f0 2024-03-25T19:38:07.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.496+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237785085 0 0) 0x7f49ec0b28a0 con 0x7f49ec1430c0 2024-03-25T19:38:07.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.496+0000 7f49e959a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.496+0000 7f49e959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.496+0000 7f49e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.496+0000 7f49e959a640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:38:07.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.496+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ec142520 con 0x7f49ec1430c0 2024-03-25T19:38:07.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.496+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104667759 0 0) 0x7f49ec0b2a30 con 0x7f49ec13f710 2024-03-25T19:38:07.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.496+0000 7f49e959a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.496+0000 7f49e959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.496+0000 7f49e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.496+0000 7f49e959a640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:38:07.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.496+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ec0b28a0 con 0x7f49ec13f710 2024-03-25T19:38:07.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3034269995 0 0) 0x7f49ec0b28a0 con 0x7f49ec13f710 2024-03-25T19:38:07.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 >> v1:172.21.15.188:6789/0 conn(0x7f49ec1430c0 legacy=0x7f49ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 >> v1:172.21.15.19:6789/0 conn(0x7f49ec0a98f0 legacy=0x7f49ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 10 monclient: _finish_hunting 0 2024-03-25T19:38:07.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 1 monclient: found mon.noname-b 2024-03-25T19:38:07.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:07.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec0b44c0 con 0x7f49ec13f710 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 10 monclient: _finish_auth 0 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 10 monclient: _check_auth_tickets 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f49e00015b0 con 0x7f49ec13f710 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49e959a640 10 monclient: dump: 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:07.512 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:07.513 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:07.513 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:07.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49f0e81640 5 monclient: authenticate success, global_id 4604 2024-03-25T19:38:07.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.511+0000 7f49f0e81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:07.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.521+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49e0001710 con 0x7f49ec13f710 2024-03-25T19:38:07.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.521+0000 7f49e959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:07.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.521+0000 7f49e959a640 1 -- 172.21.15.19:0/2221390617 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f49e0001b20 con 0x7f49ec13f710 2024-03-25T19:38:07.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.521+0000 7f49f0e81640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.521+0000 7f49f0e81640 4 set_mon_vals no callback set 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.521+0000 7f49e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.521+0000 7f49e959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.521+0000 7f49e959a640 10 monclient: dump: 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:07.522 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.522+0000 7f49f0e81640 10 monclient: shutdown 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.522+0000 7f49f0e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.522+0000 7f49f0e81640 1 -- 172.21.15.19:0/2221390617 >> v1:172.21.15.19:6790/0 conn(0x7f49ec13f710 legacy=0x7f49ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.522+0000 7f49f0e81640 1 -- 172.21.15.19:0/2221390617 shutdown_connections 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.522+0000 7f49f0e81640 1 -- 172.21.15.19:0/2221390617 >> 172.21.15.19:0/2221390617 conn(0x7f49ec0b7ea0 msgr2=0x7f49ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.522+0000 7f49f0e81640 1 -- 172.21.15.19:0/2221390617 shutdown_connections 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.522+0000 7f49f0e81640 1 -- 172.21.15.19:0/2221390617 wait complete. 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.523+0000 7f49f0e81640 10 monclient: build_initial_monmap 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.523+0000 7f49f0e81640 10 monclient: monmap: 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:38:07.524 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.524+0000 7f49f0e81640 1 librados: starting msgr at 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.524+0000 7f49f0e81640 1 librados: starting objecter 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.524+0000 7f49f0e81640 1 Processor -- start 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.524+0000 7f49f0e81640 1 -- start start 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.524+0000 7f49f0e81640 1 librados: setting wanted keys 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.524+0000 7f49f0e81640 1 librados: calling monclient init 2024-03-25T19:38:07.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.524+0000 7f49f0e81640 10 monclient: init 2024-03-25T19:38:07.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.524+0000 7f49f0e81640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:07.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.524+0000 7f49f0e81640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:07.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.524+0000 7f49f0e81640 10 monclient(hunting): picked mon.noname-a con 0x7f49ec0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:07.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.525+0000 7f49f0e81640 10 monclient(hunting): picked mon.noname-c con 0x7f49ec13f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:07.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.525+0000 7f49f0e81640 10 monclient(hunting): picked mon.noname-b con 0x7f49ec1430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:07.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.525+0000 7f49f0e81640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ec1db6a0 con 0x7f49ec0a98f0 2024-03-25T19:38:07.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.525+0000 7f49f0e81640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ec0abd30 con 0x7f49ec1430c0 2024-03-25T19:38:07.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.525+0000 7f49f0e81640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ec137f30 con 0x7f49ec13f710 2024-03-25T19:38:07.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.525+0000 7f49f0e81640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:07.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.525+0000 7f49f0e81640 10 monclient(hunting): authenticate will time out at 6540.795075s 2024-03-25T19:38:07.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.535+0000 7f49ead9d640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f49ec1430c0 0x7f49ec1e1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:58834/0 (socket says 172.21.15.19:58834) 2024-03-25T19:38:07.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.535+0000 7f49ead9d640 1 -- 172.21.15.19:0/1267168691 learned_addr learned my addr 172.21.15.19:0/1267168691 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:07.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.537+0000 7f49daffd640 10 client.?.objecter ms_handle_connect 0x7f49ec1430c0 2024-03-25T19:38:07.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.537+0000 7f49daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:07.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.542+0000 7f49daffd640 10 client.?.objecter ms_handle_connect 0x7f49ec0a98f0 2024-03-25T19:38:07.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.542+0000 7f49daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:07.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.543+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399862301 0 0) 0x7f49ec0abd30 con 0x7f49ec1430c0 2024-03-25T19:38:07.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.544+0000 7f49daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.544+0000 7f49daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.544+0000 7f49daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.544+0000 7f49daffd640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:38:07.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.544+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49cc004eb0 con 0x7f49ec1430c0 2024-03-25T19:38:07.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.547+0000 7f49daffd640 10 client.?.objecter ms_handle_connect 0x7f49ec13f710 2024-03-25T19:38:07.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.547+0000 7f49daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:07.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3924949618 0 0) 0x7f49cc004eb0 con 0x7f49ec1430c0 2024-03-25T19:38:07.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49ec0abd30 con 0x7f49ec1430c0 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f49e4001c90 con 0x7f49ec1430c0 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 10 monclient(hunting): dump: 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:07.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177705245 0 0) 0x7f49ec1db6a0 con 0x7f49ec0a98f0 2024-03-25T19:38:07.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:38:07.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.550+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49cc004eb0 con 0x7f49ec0a98f0 2024-03-25T19:38:07.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.556+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235012591 0 0) 0x7f49ec137f30 con 0x7f49ec13f710 2024-03-25T19:38:07.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.556+0000 7f49daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:07.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.556+0000 7f49daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:07.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.556+0000 7f49daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:07.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.556+0000 7f49daffd640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:38:07.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.556+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ec1db6a0 con 0x7f49ec13f710 2024-03-25T19:38:07.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2509530632 0 0) 0x7f49ec0abd30 con 0x7f49ec1430c0 2024-03-25T19:38:07.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 >> v1:172.21.15.19:6790/0 conn(0x7f49ec13f710 legacy=0x7f49ec138530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 >> v1:172.21.15.19:6789/0 conn(0x7f49ec0a98f0 legacy=0x7f49ec1daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49daffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:07.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49daffd640 1 monclient: found mon.b 2024-03-25T19:38:07.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:07.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49daffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:07.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec138270 con 0x7f49ec1430c0 2024-03-25T19:38:07.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49daffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:07.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49daffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:07.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:07.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49f0e81640 5 monclient: authenticate success, global_id 4630 2024-03-25T19:38:07.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49f0e81640 10 monclient: _renew_subs 2024-03-25T19:38:07.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49f0e81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:07.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49f0e81640 1 -- 172.21.15.19:0/1267168691 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ec138c40 con 0x7f49ec1430c0 2024-03-25T19:38:07.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49f0e81640 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:07.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49f0e81640 10 monclient: _renew_subs 2024-03-25T19:38:07.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49f0e81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:07.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49f0e81640 1 -- 172.21.15.19:0/1267168691 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ec137bd0 con 0x7f49ec1430c0 2024-03-25T19:38:07.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.558+0000 7f49f0e81640 1 librados: init done 2024-03-25T19:38:07.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.559+0000 7f49f0e81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:07.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.559+0000 7f49f0e81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:07.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.559+0000 7f49f0e81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:07.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.559+0000 7f49f0e81640 1 -- 172.21.15.19:0/1267168691 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ec1380c0 con 0x7f49ec1430c0 2024-03-25T19:38:07.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.837+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49e4003990 con 0x7f49ec1430c0 2024-03-25T19:38:07.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.837+0000 7f49daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:07.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.837+0000 7f49e8d99640 4 set_mon_vals no callback set 2024-03-25T19:38:07.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.837+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f49e4003da0 con 0x7f49ec1430c0 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.837+0000 7f49daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.837+0000 7f49daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.837+0000 7f49daffd640 10 monclient: dump: 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:07.838 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:07.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.838+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f49e4003fc0 con 0x7f49ec1430c0 2024-03-25T19:38:07.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.838+0000 7f49ea59c640 10 monclient: get_auth_request con 0x7f49cc06de80 auth_method 0 2024-03-25T19:38:07.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.838+0000 7f49daffd640 10 client.4630.objecter ms_handle_connect 0x7f49cc06de80 2024-03-25T19:38:07.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.846+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f49e4054bb0 con 0x7f49ec1430c0 2024-03-25T19:38:07.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.846+0000 7f49daffd640 10 client.4630.objecter ms_dispatch 0x7f49ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:07.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.846+0000 7f49daffd640 3 client.4630.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:07.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.846+0000 7f49daffd640 3 client.4630.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:07.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.846+0000 7f49daffd640 20 client.4630.objecter dump_active .. 0 homeless 2024-03-25T19:38:07.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.847+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49e40506c0 con 0x7f49ec1430c0 2024-03-25T19:38:07.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.847+0000 7f49daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:07.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:07.847+0000 7f49daffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:08.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.092+0000 7f49f0e81640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:08.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.092+0000 7f49f0e81640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:08.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.092+0000 7f49f0e81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:08.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.092+0000 7f49f0e81640 1 -- 172.21.15.19:0/1267168691 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49ec0627b0 con 0x7f49ec1430c0 2024-03-25T19:38:08.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.141+0000 7f49daffd640 1 -- 172.21.15.19:0/1267168691 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f49e40525f0 con 0x7f49ec1430c0 2024-03-25T19:38:08.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.141+0000 7f49daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:08.141 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.141+0000 7f49daffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:08.141 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:38:08.141 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.143+0000 7f49f0e81640 10 librados: watch_flush enter 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.143+0000 7f49f0e81640 10 librados: watch_flush exit 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.143+0000 7f49f0e81640 20 client.4630.objecter shutdown clearing up homeless session... 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.143+0000 7f49f0e81640 10 client.4630.objecter successfully canceled tick 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.143+0000 7f49f0e81640 1 -- 172.21.15.19:0/1267168691 >> v1:172.21.15.188:6800/2207506229 conn(0x7f49cc06de80 legacy=0x7f49cc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.143+0000 7f49f0e81640 10 monclient: shutdown 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.143+0000 7f49f0e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.143+0000 7f49f0e81640 1 -- 172.21.15.19:0/1267168691 >> v1:172.21.15.188:6789/0 conn(0x7f49ec1430c0 legacy=0x7f49ec1e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.143+0000 7f49f0e81640 1 -- 172.21.15.19:0/1267168691 shutdown_connections 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.143+0000 7f49f0e81640 1 -- 172.21.15.19:0/1267168691 >> 172.21.15.19:0/1267168691 conn(0x7f49ec0b7ea0 msgr2=0x7f49ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.143+0000 7f49f0e81640 1 -- 172.21.15.19:0/1267168691 shutdown_connections 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.143+0000 7f49f0e81640 1 -- 172.21.15.19:0/1267168691 wait complete. 2024-03-25T19:38:08.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:08.144+0000 7f49f0e81640 1 librados: shutdown 2024-03-25T19:38:08.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:14.961 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:15.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.059+0000 7f05844fb640 10 monclient: get_monmap_and_config 2024-03-25T19:38:15.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.059+0000 7f05844fb640 10 monclient: build_initial_monmap 2024-03-25T19:38:15.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.059+0000 7f05844fb640 10 monclient: monmap: 2024-03-25T19:38:15.059 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:15.059 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:15.059 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:38:15.060389+0000 2024-03-25T19:38:15.059 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:38:15.060389+0000 2024-03-25T19:38:15.059 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:15.059 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:15.059 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:15.060 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:38:15.060 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:38:15.060 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.059+0000 7f05844fb640 1 Processor -- start 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.060+0000 7f05844fb640 1 -- start start 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.060+0000 7f05844fb640 10 monclient: init 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.061+0000 7f05844fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.061+0000 7f05844fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.061+0000 7f05844fb640 10 monclient(hunting): picked mon.noname-a con 0x7f057c0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.061+0000 7f05844fb640 10 monclient(hunting): picked mon.noname-c con 0x7f057c13f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.061+0000 7f05844fb640 10 monclient(hunting): picked mon.noname-b con 0x7f057c1430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.061+0000 7f05844fb640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c142520 con 0x7f057c0a98f0 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.061+0000 7f05844fb640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c0b28a0 con 0x7f057c13f710 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.061+0000 7f05844fb640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c0b2a30 con 0x7f057c1430c0 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.061+0000 7f05844fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:15.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.061+0000 7f05844fb640 10 monclient(hunting): authenticate will time out at 6548.331052s 2024-03-25T19:38:15.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.067+0000 7f0582297640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f057c0a98f0 0x7f057c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:41294/0 (socket says 172.21.15.19:41294) 2024-03-25T19:38:15.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.067+0000 7f0582297640 1 -- 172.21.15.19:0/763959214 learned_addr learned my addr 172.21.15.19:0/763959214 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:15.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.077+0000 7f0581295640 1 -- 172.21.15.19:0/763959214 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518366423 0 0) 0x7f057c0b28a0 con 0x7f057c13f710 2024-03-25T19:38:15.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.077+0000 7f0581295640 10 monclient(hunting): _negotiate 2024-03-25T19:38:15.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.077+0000 7f0581295640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:15.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.077+0000 7f0581295640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:15.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.077+0000 7f0581295640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:38:15.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.077+0000 7f0581295640 1 -- 172.21.15.19:0/763959214 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0568003520 con 0x7f057c13f710 2024-03-25T19:38:15.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.079+0000 7f0581295640 1 -- 172.21.15.19:0/763959214 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151028862 0 0) 0x7f057c142520 con 0x7f057c0a98f0 2024-03-25T19:38:15.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.079+0000 7f0581295640 10 monclient(hunting): _negotiate 2024-03-25T19:38:15.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.079+0000 7f0581295640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:15.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.079+0000 7f0581295640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:15.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.079+0000 7f0581295640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:38:15.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.079+0000 7f0581295640 1 -- 172.21.15.19:0/763959214 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f057c0b28a0 con 0x7f057c0a98f0 2024-03-25T19:38:15.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.094+0000 7f0581295640 1 -- 172.21.15.19:0/763959214 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 86557700 0 0) 0x7f0568003520 con 0x7f057c13f710 2024-03-25T19:38:15.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.094+0000 7f0581295640 1 -- 172.21.15.19:0/763959214 >> v1:172.21.15.19:6790/0 conn(0x7f057c1430c0 legacy=0x7f057c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:38:15.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 1 -- 172.21.15.19:0/763959214 >> v1:172.21.15.19:6789/0 conn(0x7f057c0a98f0 legacy=0x7f057c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 10 monclient: _finish_hunting 0 2024-03-25T19:38:15.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 1 monclient: found mon.noname-c 2024-03-25T19:38:15.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:15.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:38:15.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 1 -- 172.21.15.19:0/763959214 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c0b44c0 con 0x7f057c13f710 2024-03-25T19:38:15.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 10 monclient: _finish_auth 0 2024-03-25T19:38:15.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 10 monclient: _check_auth_tickets 2024-03-25T19:38:15.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 1 -- 172.21.15.19:0/763959214 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f056c001690 con 0x7f057c13f710 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f0581295640 10 monclient: dump: 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f05844fb640 5 monclient: authenticate success, global_id 4633 2024-03-25T19:38:15.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.095+0000 7f05844fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:15.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.119+0000 7f0581295640 1 -- 172.21.15.19:0/763959214 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f056c0017f0 con 0x7f057c13f710 2024-03-25T19:38:15.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.119+0000 7f0581295640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:15.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.119+0000 7f0581295640 1 -- 172.21.15.19:0/763959214 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f056c001c00 con 0x7f057c13f710 2024-03-25T19:38:15.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.119+0000 7f0581295640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:15.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.119+0000 7f0581295640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:15.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.119+0000 7f0581295640 10 monclient: dump: 2024-03-25T19:38:15.121 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:15.121 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:15.121 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:15.121 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:15.121 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:15.121 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:15.121 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:15.121 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:15.121 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:15.121 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:15.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.119+0000 7f05844fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:15.122 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.119+0000 7f05844fb640 4 set_mon_vals no callback set 2024-03-25T19:38:15.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.120+0000 7f05844fb640 10 monclient: shutdown 2024-03-25T19:38:15.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.120+0000 7f05844fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:15.123 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.120+0000 7f05844fb640 1 -- 172.21.15.19:0/763959214 >> v1:172.21.15.188:6789/0 conn(0x7f057c13f710 legacy=0x7f057c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.120+0000 7f05844fb640 1 -- 172.21.15.19:0/763959214 shutdown_connections 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.120+0000 7f05844fb640 1 -- 172.21.15.19:0/763959214 >> 172.21.15.19:0/763959214 conn(0x7f057c0b7ea0 msgr2=0x7f057c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.120+0000 7f05844fb640 1 -- 172.21.15.19:0/763959214 shutdown_connections 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.120+0000 7f05844fb640 1 -- 172.21.15.19:0/763959214 wait complete. 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.121+0000 7f05844fb640 10 monclient: build_initial_monmap 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.121+0000 7f05844fb640 10 monclient: monmap: 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:38:15.124 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.121+0000 7f05844fb640 1 librados: starting msgr at 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.121+0000 7f05844fb640 1 librados: starting objecter 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.121+0000 7f05844fb640 1 Processor -- start 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.121+0000 7f05844fb640 1 -- start start 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.121+0000 7f05844fb640 1 librados: setting wanted keys 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.121+0000 7f05844fb640 1 librados: calling monclient init 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.121+0000 7f05844fb640 10 monclient: init 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.122+0000 7f05844fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.122+0000 7f05844fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.122+0000 7f05844fb640 10 monclient(hunting): picked mon.noname-b con 0x7f057c0a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.122+0000 7f05844fb640 10 monclient(hunting): picked mon.noname-a con 0x7f057c13f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.122+0000 7f05844fb640 10 monclient(hunting): picked mon.noname-c con 0x7f057c1430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.122+0000 7f05844fb640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c136a30 con 0x7f057c13f710 2024-03-25T19:38:15.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.122+0000 7f05844fb640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c136bc0 con 0x7f057c0a98f0 2024-03-25T19:38:15.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.122+0000 7f05844fb640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c136d50 con 0x7f057c1430c0 2024-03-25T19:38:15.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.122+0000 7f05844fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:15.126 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.122+0000 7f05844fb640 10 monclient(hunting): authenticate will time out at 6548.392513s 2024-03-25T19:38:15.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.129+0000 7f0582297640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f057c0a98f0 0x7f057c1dab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:45870/0 (socket says 172.21.15.19:45870) 2024-03-25T19:38:15.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.129+0000 7f0582297640 1 -- 172.21.15.19:0/1131989528 learned_addr learned my addr 172.21.15.19:0/1131989528 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:15.133 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.133+0000 7f0572ffd640 10 client.?.objecter ms_handle_connect 0x7f057c0a98f0 2024-03-25T19:38:15.134 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.133+0000 7f0572ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:15.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.139+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956940845 0 0) 0x7f057c136bc0 con 0x7f057c0a98f0 2024-03-25T19:38:15.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.139+0000 7f0572ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:15.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.139+0000 7f0572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:15.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.139+0000 7f0572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:15.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.139+0000 7f0572ffd640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:38:15.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.139+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0560004eb0 con 0x7f057c0a98f0 2024-03-25T19:38:15.147 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.147+0000 7f0572ffd640 10 client.?.objecter ms_handle_connect 0x7f057c13f710 2024-03-25T19:38:15.147 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.147+0000 7f0572ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:15.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.149+0000 7f0572ffd640 10 client.?.objecter ms_handle_connect 0x7f057c1430c0 2024-03-25T19:38:15.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.149+0000 7f0572ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:15.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.152+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2696279443 0 0) 0x7f0560004eb0 con 0x7f057c0a98f0 2024-03-25T19:38:15.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.152+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f057c136bc0 con 0x7f057c0a98f0 2024-03-25T19:38:15.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.152+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f05780019f0 con 0x7f057c0a98f0 2024-03-25T19:38:15.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.152+0000 7f0572ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:15.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.152+0000 7f0572ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:15.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.152+0000 7f0572ffd640 10 monclient(hunting): dump: 2024-03-25T19:38:15.154 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:15.154 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:15.154 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:15.154 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:15.154 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:15.154 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:15.154 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:15.154 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:15.154 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:15.154 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:15.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.152+0000 7f0572ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:15.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.161+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269011962 0 0) 0x7f057c136a30 con 0x7f057c13f710 2024-03-25T19:38:15.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.161+0000 7f0572ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:15.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.161+0000 7f0572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:15.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.161+0000 7f0572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:15.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.161+0000 7f0572ffd640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:38:15.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.161+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0560004eb0 con 0x7f057c13f710 2024-03-25T19:38:15.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1333735360 0 0) 0x7f057c136bc0 con 0x7f057c0a98f0 2024-03-25T19:38:15.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 >> v1:172.21.15.19:6790/0 conn(0x7f057c1430c0 legacy=0x7f057c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 >> v1:172.21.15.19:6789/0 conn(0x7f057c13f710 legacy=0x7f057c137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f0572ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:15.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f0572ffd640 1 monclient: found mon.b 2024-03-25T19:38:15.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f0572ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:15.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f0572ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:15.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c0b7110 con 0x7f057c0a98f0 2024-03-25T19:38:15.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f0572ffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:15.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f0572ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:15.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f0572ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:15.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f05844fb640 5 monclient: authenticate success, global_id 4636 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f05844fb640 10 monclient: _renew_subs 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f05844fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f05844fb640 1 -- 172.21.15.19:0/1131989528 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057c0b7380 con 0x7f057c0a98f0 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f05844fb640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f05844fb640 10 monclient: _renew_subs 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f05844fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:15.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f05844fb640 1 -- 172.21.15.19:0/1131989528 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f057c139600 con 0x7f057c0a98f0 2024-03-25T19:38:15.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.162+0000 7f05844fb640 1 librados: init done 2024-03-25T19:38:15.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.163+0000 7f05844fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:15.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.163+0000 7f05844fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:15.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.163+0000 7f05844fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:15.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.163+0000 7f05844fb640 1 -- 172.21.15.19:0/1131989528 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057c0b3ef0 con 0x7f057c0a98f0 2024-03-25T19:38:15.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.453+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0578001c70 con 0x7f057c0a98f0 2024-03-25T19:38:15.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.453+0000 7f0572ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:15.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.453+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f0578004c30 con 0x7f057c0a98f0 2024-03-25T19:38:15.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.453+0000 7f0572ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:15.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.453+0000 7f0572ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:15.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.453+0000 7f0572ffd640 10 monclient: dump: 2024-03-25T19:38:15.454 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:15.454 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:15.454 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:15.454 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:15.454 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:15.454 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:15.454 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:15.454 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:15.454 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:15.454 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:15.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.453+0000 7f0580a94640 4 set_mon_vals no callback set 2024-03-25T19:38:15.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.454+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f057801ac00 con 0x7f057c0a98f0 2024-03-25T19:38:15.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.454+0000 7f0581a96640 10 monclient: get_auth_request con 0x7f056006dd90 auth_method 0 2024-03-25T19:38:15.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.455+0000 7f0572ffd640 10 client.4636.objecter ms_handle_connect 0x7f056006dd90 2024-03-25T19:38:15.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.458+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f0578054810 con 0x7f057c0a98f0 2024-03-25T19:38:15.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.458+0000 7f0572ffd640 10 client.4636.objecter ms_dispatch 0x7f057c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:15.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.458+0000 7f0572ffd640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:15.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.458+0000 7f0572ffd640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:15.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.458+0000 7f0572ffd640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T19:38:15.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.459+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0578050150 con 0x7f057c0a98f0 2024-03-25T19:38:15.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.459+0000 7f0572ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:15.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.459+0000 7f0572ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:15.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.725+0000 7f05844fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:15.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.725+0000 7f05844fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:15.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.725+0000 7f05844fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:15.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.725+0000 7f05844fb640 1 -- 172.21.15.19:0/1131989528 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f057c0627b0 con 0x7f057c0a98f0 2024-03-25T19:38:15.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.750+0000 7f0572ffd640 1 -- 172.21.15.19:0/1131989528 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f0578052250 con 0x7f057c0a98f0 2024-03-25T19:38:15.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.750+0000 7f0572ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:15.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.750+0000 7f0572ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:15.751 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:38:15.751 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:15.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 10 librados: watch_flush enter 2024-03-25T19:38:15.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 10 librados: watch_flush exit 2024-03-25T19:38:15.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T19:38:15.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 10 client.4636.objecter successfully canceled tick 2024-03-25T19:38:15.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 1 -- 172.21.15.19:0/1131989528 >> v1:172.21.15.188:6800/2207506229 conn(0x7f056006dd90 legacy=0x7f0560070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 10 monclient: shutdown 2024-03-25T19:38:15.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:15.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 1 -- 172.21.15.19:0/1131989528 >> v1:172.21.15.188:6789/0 conn(0x7f057c0a98f0 legacy=0x7f057c1dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 1 -- 172.21.15.19:0/1131989528 shutdown_connections 2024-03-25T19:38:15.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 1 -- 172.21.15.19:0/1131989528 >> 172.21.15.19:0/1131989528 conn(0x7f057c0b7ea0 msgr2=0x7f057c13c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:15.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 1 -- 172.21.15.19:0/1131989528 shutdown_connections 2024-03-25T19:38:15.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 1 -- 172.21.15.19:0/1131989528 wait complete. 2024-03-25T19:38:15.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:15.753+0000 7f05844fb640 1 librados: shutdown 2024-03-25T19:38:15.765 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:18.421 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:41.151 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.432 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:41.162 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.479 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:41.208 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.483 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:09:41.186 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.489 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:09:41.192 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.498 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:41.227 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.506 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:41.236 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.520 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:09:41.223 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.542 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:41.272 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:22.571 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:22.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.685+0000 7f4b3117b640 10 monclient: get_monmap_and_config 2024-03-25T19:38:22.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.685+0000 7f4b3117b640 10 monclient: build_initial_monmap 2024-03-25T19:38:22.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.685+0000 7f4b3117b640 10 monclient: monmap: 2024-03-25T19:38:22.685 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:22.686 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:22.686 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:38:22.686690+0000 2024-03-25T19:38:22.686 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:38:22.686690+0000 2024-03-25T19:38:22.686 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:22.686 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:22.686 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:22.686 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:38:22.686 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:38:22.686 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:22.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 1 Processor -- start 2024-03-25T19:38:22.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 1 -- start start 2024-03-25T19:38:22.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 10 monclient: init 2024-03-25T19:38:22.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:22.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:22.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 10 monclient(hunting): picked mon.noname-b con 0x7f4b2c0b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:22.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 10 monclient(hunting): picked mon.noname-a con 0x7f4b2c0b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:22.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 10 monclient(hunting): picked mon.noname-c con 0x7f4b2c14be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:22.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2c0af940 con 0x7f4b2c0b1730 2024-03-25T19:38:22.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2c0b0a70 con 0x7f4b2c14be90 2024-03-25T19:38:22.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2c0b0d50 con 0x7f4b2c0b4d60 2024-03-25T19:38:22.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:22.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.686+0000 7f4b3117b640 10 monclient(hunting): authenticate will time out at 6555.956832s 2024-03-25T19:38:22.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.688+0000 7f4b2b59e640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f4b2c14be90 0x7f4b2c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:58258/0 (socket says 172.21.15.19:58258) 2024-03-25T19:38:22.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.688+0000 7f4b2b59e640 1 -- 172.21.15.19:0/3762640466 learned_addr learned my addr 172.21.15.19:0/3762640466 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:22.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.705+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396367845 0 0) 0x7f4b2c0b0a70 con 0x7f4b2c14be90 2024-03-25T19:38:22.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.705+0000 7f4b29d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.705+0000 7f4b29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.705+0000 7f4b29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.705+0000 7f4b29d9b640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:38:22.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.705+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b14003520 con 0x7f4b2c14be90 2024-03-25T19:38:22.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.717+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893082729 0 0) 0x7f4b2c0b0d50 con 0x7f4b2c0b4d60 2024-03-25T19:38:22.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.717+0000 7f4b29d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.717+0000 7f4b29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.717+0000 7f4b29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.717+0000 7f4b29d9b640 10 monclient(hunting): my global_id is 4613 2024-03-25T19:38:22.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.717+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b2c0b0a70 con 0x7f4b2c0b4d60 2024-03-25T19:38:22.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.724+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734970380 0 0) 0x7f4b2c0af940 con 0x7f4b2c0b1730 2024-03-25T19:38:22.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.724+0000 7f4b29d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.724+0000 7f4b29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.724+0000 7f4b29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.724+0000 7f4b29d9b640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:38:22.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.724+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b2c0b0d50 con 0x7f4b2c0b1730 2024-03-25T19:38:22.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2202759895 0 0) 0x7f4b2c0b0a70 con 0x7f4b2c0b4d60 2024-03-25T19:38:22.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 >> v1:172.21.15.188:6789/0 conn(0x7f4b2c14be90 legacy=0x7f4b2c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 >> v1:172.21.15.19:6789/0 conn(0x7f4b2c0b1730 legacy=0x7f4b2c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:38:22.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 1 monclient: found mon.noname-b 2024-03-25T19:38:22.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:22.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c0b1030 con 0x7f4b2c0b4d60 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 10 monclient: _finish_auth 0 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f4b1c001670 con 0x7f4b2c0b4d60 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b29d9b640 10 monclient: dump: 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.728 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:22.728 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:22.728 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:22.728 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:22.728 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:22.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b3117b640 5 monclient: authenticate success, global_id 4613 2024-03-25T19:38:22.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.725+0000 7f4b3117b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:22.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.737+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b1c0017d0 con 0x7f4b2c0b4d60 2024-03-25T19:38:22.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.737+0000 7f4b29d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:22.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.737+0000 7f4b29d9b640 1 -- 172.21.15.19:0/3762640466 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f4b1c001ba0 con 0x7f4b2c0b4d60 2024-03-25T19:38:22.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.737+0000 7f4b29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.737+0000 7f4b29d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:22.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.737+0000 7f4b29d9b640 10 monclient: dump: 2024-03-25T19:38:22.738 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:22.738 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:22.739 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:22.739 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:22.739 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.739 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:22.739 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:22.740 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:22.740 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:22.740 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:22.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.737+0000 7f4b3117b640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:22.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.737+0000 7f4b3117b640 4 set_mon_vals no callback set 2024-03-25T19:38:22.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.738+0000 7f4b3117b640 10 monclient: shutdown 2024-03-25T19:38:22.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.738+0000 7f4b3117b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:22.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.738+0000 7f4b3117b640 1 -- 172.21.15.19:0/3762640466 >> v1:172.21.15.19:6790/0 conn(0x7f4b2c0b4d60 legacy=0x7f4b2c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.738+0000 7f4b3117b640 1 -- 172.21.15.19:0/3762640466 shutdown_connections 2024-03-25T19:38:22.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.738+0000 7f4b3117b640 1 -- 172.21.15.19:0/3762640466 >> 172.21.15.19:0/3762640466 conn(0x7f4b2c0617a0 msgr2=0x7f4b2c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:22.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.738+0000 7f4b3117b640 1 -- 172.21.15.19:0/3762640466 shutdown_connections 2024-03-25T19:38:22.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.738+0000 7f4b3117b640 1 -- 172.21.15.19:0/3762640466 wait complete. 2024-03-25T19:38:22.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.739+0000 7f4b3117b640 10 monclient: build_initial_monmap 2024-03-25T19:38:22.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.739+0000 7f4b3117b640 10 monclient: monmap: 2024-03-25T19:38:22.743 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:22.743 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:22.743 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.739+0000 7f4b3117b640 1 librados: starting msgr at 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.739+0000 7f4b3117b640 1 librados: starting objecter 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.740+0000 7f4b3117b640 1 Processor -- start 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.740+0000 7f4b3117b640 1 -- start start 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.740+0000 7f4b3117b640 1 librados: setting wanted keys 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.740+0000 7f4b3117b640 1 librados: calling monclient init 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.740+0000 7f4b3117b640 10 monclient: init 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.740+0000 7f4b3117b640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.740+0000 7f4b3117b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:22.744 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.740+0000 7f4b3117b640 10 monclient(hunting): picked mon.noname-b con 0x7f4b2c0b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:22.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.741+0000 7f4b3117b640 10 monclient(hunting): picked mon.noname-a con 0x7f4b2c0b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:22.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.741+0000 7f4b3117b640 10 monclient(hunting): picked mon.noname-c con 0x7f4b2c14be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:22.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.741+0000 7f4b3117b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2c1d7d30 con 0x7f4b2c0b4d60 2024-03-25T19:38:22.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.741+0000 7f4b3117b640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2c1429f0 con 0x7f4b2c0b1730 2024-03-25T19:38:22.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.741+0000 7f4b3117b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2c137b20 con 0x7f4b2c14be90 2024-03-25T19:38:22.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.741+0000 7f4b3117b640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:22.745 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.741+0000 7f4b3117b640 10 monclient(hunting): authenticate will time out at 6556.010992s 2024-03-25T19:38:22.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.751+0000 7f4b2b59e640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f4b2c14be90 0x7f4b2c1e2e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:45966/0 (socket says 172.21.15.19:45966) 2024-03-25T19:38:22.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.751+0000 7f4b2b59e640 1 -- 172.21.15.19:0/4126993771 learned_addr learned my addr 172.21.15.19:0/4126993771 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:22.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.752+0000 7f4b137fe640 10 client.?.objecter ms_handle_connect 0x7f4b2c14be90 2024-03-25T19:38:22.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.752+0000 7f4b137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:22.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.754+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442905963 0 0) 0x7f4b2c137b20 con 0x7f4b2c14be90 2024-03-25T19:38:22.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.754+0000 7f4b137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.754+0000 7f4b137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.754+0000 7f4b137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.754+0000 7f4b137fe640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:38:22.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.754+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b18004eb0 con 0x7f4b2c14be90 2024-03-25T19:38:22.763 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.764+0000 7f4b137fe640 10 client.?.objecter ms_handle_connect 0x7f4b2c0b4d60 2024-03-25T19:38:22.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.764+0000 7f4b137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:22.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.773+0000 7f4b137fe640 10 client.?.objecter ms_handle_connect 0x7f4b2c0b1730 2024-03-25T19:38:22.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.773+0000 7f4b137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:22.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.773+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2905339441 0 0) 0x7f4b18004eb0 con 0x7f4b2c14be90 2024-03-25T19:38:22.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.773+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b2c137b20 con 0x7f4b2c14be90 2024-03-25T19:38:22.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.773+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f4b24001b30 con 0x7f4b2c14be90 2024-03-25T19:38:22.774 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.773+0000 7f4b137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.773+0000 7f4b137fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:22.775 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.773+0000 7f4b137fe640 10 monclient(hunting): dump: 2024-03-25T19:38:22.775 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:22.775 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:22.775 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:22.775 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:22.775 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.775 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:22.775 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:22.775 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:22.775 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:22.775 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:22.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.773+0000 7f4b137fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:22.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2695974335 0 0) 0x7f4b2c137b20 con 0x7f4b2c14be90 2024-03-25T19:38:22.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 >> v1:172.21.15.188:6789/0 conn(0x7f4b2c0b1730 legacy=0x7f4b2c1d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 >> v1:172.21.15.19:6789/0 conn(0x7f4b2c0b4d60 legacy=0x7f4b2c137410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b137fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:22.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b137fe640 1 monclient: found mon.c 2024-03-25T19:38:22.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:22.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b137fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:38:22.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c140eb0 con 0x7f4b2c14be90 2024-03-25T19:38:22.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b137fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:22.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b137fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:22.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:22.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b3117b640 5 monclient: authenticate success, global_id 4616 2024-03-25T19:38:22.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b3117b640 10 monclient: _renew_subs 2024-03-25T19:38:22.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b3117b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:38:22.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b3117b640 1 -- 172.21.15.19:0/4126993771 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b2c142610 con 0x7f4b2c14be90 2024-03-25T19:38:22.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b3117b640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:22.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b3117b640 10 monclient: _renew_subs 2024-03-25T19:38:22.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b3117b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:38:22.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b3117b640 1 -- 172.21.15.19:0/4126993771 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b2c1385c0 con 0x7f4b2c14be90 2024-03-25T19:38:22.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.778+0000 7f4b3117b640 1 librados: init done 2024-03-25T19:38:22.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.780+0000 7f4b3117b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:22.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.780+0000 7f4b3117b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:22.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.780+0000 7f4b3117b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:38:22.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:22.780+0000 7f4b3117b640 1 -- 172.21.15.19:0/4126993771 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b2c142850 con 0x7f4b2c14be90 2024-03-25T19:38:23.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.096+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b240020a0 con 0x7f4b2c14be90 2024-03-25T19:38:23.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.096+0000 7f4b137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:23.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.096+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f4b24004b60 con 0x7f4b2c14be90 2024-03-25T19:38:23.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.096+0000 7f4b137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:23.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.096+0000 7f4b137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:23.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.096+0000 7f4b137fe640 10 monclient: dump: 2024-03-25T19:38:23.097 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:23.097 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:23.098 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:23.098 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:23.098 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:23.098 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:23.098 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:23.098 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:23.099 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:23.099 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:23.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.096+0000 7f4b2959a640 4 set_mon_vals no callback set 2024-03-25T19:38:23.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.097+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f4b24004d80 con 0x7f4b2c14be90 2024-03-25T19:38:23.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.098+0000 7f4b2ad9d640 10 monclient: get_auth_request con 0x7f4b1806dc40 auth_method 0 2024-03-25T19:38:23.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.098+0000 7f4b137fe640 10 client.4616.objecter ms_handle_connect 0x7f4b1806dc40 2024-03-25T19:38:23.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.103+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f4b24054990 con 0x7f4b2c14be90 2024-03-25T19:38:23.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.103+0000 7f4b137fe640 10 client.4616.objecter ms_dispatch 0x7f4b2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:23.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.103+0000 7f4b137fe640 3 client.4616.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:23.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.103+0000 7f4b137fe640 3 client.4616.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:23.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.103+0000 7f4b137fe640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-25T19:38:23.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.103+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b240502d0 con 0x7f4b2c14be90 2024-03-25T19:38:23.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.103+0000 7f4b137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:23.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.103+0000 7f4b137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:23.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.322+0000 7f4b3117b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:23.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.322+0000 7f4b3117b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:23.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.322+0000 7f4b3117b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:38:23.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.322+0000 7f4b3117b640 1 -- 172.21.15.19:0/4126993771 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b2c141040 con 0x7f4b2c14be90 2024-03-25T19:38:23.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.358+0000 7f4b137fe640 1 -- 172.21.15.19:0/4126993771 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f4b2c141040 con 0x7f4b2c14be90 2024-03-25T19:38:23.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.358+0000 7f4b137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:23.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.358+0000 7f4b137fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:23.358 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:38:23.358 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:23.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 10 librados: watch_flush enter 2024-03-25T19:38:23.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 10 librados: watch_flush exit 2024-03-25T19:38:23.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-25T19:38:23.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 10 client.4616.objecter successfully canceled tick 2024-03-25T19:38:23.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 1 -- 172.21.15.19:0/4126993771 >> v1:172.21.15.188:6800/2207506229 conn(0x7f4b1806dc40 legacy=0x7f4b18070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:23.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 10 monclient: shutdown 2024-03-25T19:38:23.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:23.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 1 -- 172.21.15.19:0/4126993771 >> v1:172.21.15.19:6790/0 conn(0x7f4b2c14be90 legacy=0x7f4b2c1e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:23.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 1 -- 172.21.15.19:0/4126993771 shutdown_connections 2024-03-25T19:38:23.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 1 -- 172.21.15.19:0/4126993771 >> 172.21.15.19:0/4126993771 conn(0x7f4b2c0617a0 msgr2=0x7f4b2c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:23.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 1 -- 172.21.15.19:0/4126993771 shutdown_connections 2024-03-25T19:38:23.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 1 -- 172.21.15.19:0/4126993771 wait complete. 2024-03-25T19:38:23.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:23.361+0000 7f4b3117b640 1 librados: shutdown 2024-03-25T19:38:23.372 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:30.178 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:30.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.292+0000 7f3bdf843640 10 monclient: get_monmap_and_config 2024-03-25T19:38:30.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.292+0000 7f3bdf843640 10 monclient: build_initial_monmap 2024-03-25T19:38:30.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.293+0000 7f3bdf843640 10 monclient: monmap: 2024-03-25T19:38:30.293 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:30.293 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:30.293 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:38:30.294226+0000 2024-03-25T19:38:30.293 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:38:30.294226+0000 2024-03-25T19:38:30.293 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:30.293 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:30.293 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:30.293 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:38:30.293 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:38:30.293 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:30.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.293+0000 7f3bdf843640 1 Processor -- start 2024-03-25T19:38:30.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.293+0000 7f3bdf843640 1 -- start start 2024-03-25T19:38:30.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.293+0000 7f3bdf843640 10 monclient: init 2024-03-25T19:38:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.294+0000 7f3bdf843640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.294+0000 7f3bdf843640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.294+0000 7f3bdf843640 10 monclient(hunting): picked mon.noname-b con 0x7f3bd813fc60 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.294+0000 7f3bdf843640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd8070570 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.294+0000 7f3bdf843640 10 monclient(hunting): picked mon.noname-a con 0x7f3bd80727a0 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.294+0000 7f3bdf843640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd813c9f0 con 0x7f3bd80727a0 2024-03-25T19:38:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.295+0000 7f3bdf843640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd806fac0 con 0x7f3bd8070570 2024-03-25T19:38:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.295+0000 7f3bdf843640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd80715d0 con 0x7f3bd813fc60 2024-03-25T19:38:30.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.295+0000 7f3bdf843640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:30.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.295+0000 7f3bdf843640 10 monclient(hunting): authenticate will time out at 6563.564944s 2024-03-25T19:38:30.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.302+0000 7f3bdd5df640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f3bd813fc60 0x7f3bd806f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:44346/0 (socket says 172.21.15.19:44346) 2024-03-25T19:38:30.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.302+0000 7f3bdd5df640 1 -- 172.21.15.19:0/2283533878 learned_addr learned my addr 172.21.15.19:0/2283533878 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:30.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.314+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155972081 0 0) 0x7f3bd813c9f0 con 0x7f3bd80727a0 2024-03-25T19:38:30.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.314+0000 7f3bcffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:30.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.314+0000 7f3bcffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:30.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.314+0000 7f3bcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:30.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.314+0000 7f3bcffff640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:38:30.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.314+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc4005730 con 0x7f3bd80727a0 2024-03-25T19:38:30.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.321+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617605968 0 0) 0x7f3bd80715d0 con 0x7f3bd813fc60 2024-03-25T19:38:30.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.321+0000 7f3bcffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:30.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.321+0000 7f3bcffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:30.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.321+0000 7f3bcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:30.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.321+0000 7f3bcffff640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:38:30.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.321+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd813c9f0 con 0x7f3bd813fc60 2024-03-25T19:38:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.322+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757100780 0 0) 0x7f3bd806fac0 con 0x7f3bd8070570 2024-03-25T19:38:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.322+0000 7f3bcffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.322+0000 7f3bcffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.322+0000 7f3bcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.322+0000 7f3bcffff640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:38:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.322+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd80715d0 con 0x7f3bd8070570 2024-03-25T19:38:30.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2361765425 0 0) 0x7f3bd813c9f0 con 0x7f3bd813fc60 2024-03-25T19:38:30.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 >> v1:172.21.15.188:6789/0 conn(0x7f3bd8070570 legacy=0x7f3bd8072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 >> v1:172.21.15.19:6789/0 conn(0x7f3bd80727a0 legacy=0x7f3bd814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 1 monclient: found mon.noname-b 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd80718b0 con 0x7f3bd813fc60 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 10 monclient: _finish_auth 0 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3bd4001670 con 0x7f3bd813fc60 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bdf843640 5 monclient: authenticate success, global_id 4619 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bdf843640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bcffff640 10 monclient: dump: 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:30.336 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:30.337 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:30.337 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:30.337 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.337 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:30.337 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:30.337 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:30.337 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:30.337 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:30.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.335+0000 7f3bdf843640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:30.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.343+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bd40017d0 con 0x7f3bd813fc60 2024-03-25T19:38:30.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.343+0000 7f3bcffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:30.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.343+0000 7f3bcffff640 1 -- 172.21.15.19:0/2283533878 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3bd4001be0 con 0x7f3bd813fc60 2024-03-25T19:38:30.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.343+0000 7f3bdf843640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:30.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.343+0000 7f3bdf843640 4 set_mon_vals no callback set 2024-03-25T19:38:30.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.343+0000 7f3bcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.343+0000 7f3bcffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:30.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.343+0000 7f3bcffff640 10 monclient: dump: 2024-03-25T19:38:30.344 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:30.344 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:30.344 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:30.344 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:30.344 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.344 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:30.344 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:30.344 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:30.344 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:30.344 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:30.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.344+0000 7f3bdf843640 10 monclient: shutdown 2024-03-25T19:38:30.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.344+0000 7f3bdf843640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:30.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.344+0000 7f3bdf843640 1 -- 172.21.15.19:0/2283533878 >> v1:172.21.15.19:6790/0 conn(0x7f3bd813fc60 legacy=0x7f3bd806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.344+0000 7f3bdf843640 1 -- 172.21.15.19:0/2283533878 shutdown_connections 2024-03-25T19:38:30.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.344+0000 7f3bdf843640 1 -- 172.21.15.19:0/2283533878 >> 172.21.15.19:0/2283533878 conn(0x7f3bd80b4a80 msgr2=0x7f3bd80b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:30.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.344+0000 7f3bdf843640 1 -- 172.21.15.19:0/2283533878 shutdown_connections 2024-03-25T19:38:30.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.344+0000 7f3bdf843640 1 -- 172.21.15.19:0/2283533878 wait complete. 2024-03-25T19:38:30.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.345+0000 7f3bdf843640 10 monclient: build_initial_monmap 2024-03-25T19:38:30.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.345+0000 7f3bdf843640 10 monclient: monmap: 2024-03-25T19:38:30.346 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:30.346 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:30.346 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:38:30.346 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:38:30.346 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:30.346 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:38:30.346 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:30.346 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:38:30.346 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:38:30.346 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:30.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.345+0000 7f3bdf843640 1 librados: starting msgr at 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.345+0000 7f3bdf843640 1 librados: starting objecter 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 1 Processor -- start 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 1 -- start start 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 1 librados: setting wanted keys 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 1 librados: calling monclient init 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 10 monclient: init 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 10 monclient(hunting): picked mon.noname-a con 0x7f3bd8070570 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 10 monclient(hunting): picked mon.noname-c con 0x7f3bd80727a0 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 10 monclient(hunting): picked mon.noname-b con 0x7f3bd813fc60 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd81db110 con 0x7f3bd8070570 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd813a810 con 0x7f3bd813fc60 2024-03-25T19:38:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.346+0000 7f3bdf843640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd81dad30 con 0x7f3bd80727a0 2024-03-25T19:38:30.349 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.347+0000 7f3bdf843640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:30.349 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.347+0000 7f3bdf843640 10 monclient(hunting): authenticate will time out at 6563.616928s 2024-03-25T19:38:30.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.357+0000 7f3bdd5df640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f3bd8070570 0x7f3bd81d7830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:55056/0 (socket says 172.21.15.19:55056) 2024-03-25T19:38:30.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.357+0000 7f3bdd5df640 1 -- 172.21.15.19:0/2878114384 learned_addr learned my addr 172.21.15.19:0/2878114384 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:30.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.357+0000 7f3bcdffb640 10 client.?.objecter ms_handle_connect 0x7f3bd8070570 2024-03-25T19:38:30.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.357+0000 7f3bcdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:30.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.358+0000 7f3bcdffb640 10 client.?.objecter ms_handle_connect 0x7f3bd80727a0 2024-03-25T19:38:30.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.358+0000 7f3bcdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:30.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.365+0000 7f3bcdffb640 10 client.?.objecter ms_handle_connect 0x7f3bd813fc60 2024-03-25T19:38:30.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.365+0000 7f3bcdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:30.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.376+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766188152 0 0) 0x7f3bd813a810 con 0x7f3bd813fc60 2024-03-25T19:38:30.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.376+0000 7f3bcdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:30.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.376+0000 7f3bcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:30.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.376+0000 7f3bcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:30.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.376+0000 7f3bcdffb640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:38:30.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.376+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bbc004eb0 con 0x7f3bd813fc60 2024-03-25T19:38:30.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.380+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873065096 0 0) 0x7f3bd81db110 con 0x7f3bd8070570 2024-03-25T19:38:30.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.380+0000 7f3bcdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:30.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.380+0000 7f3bcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:30.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.380+0000 7f3bcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:30.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.380+0000 7f3bcdffb640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:38:30.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.380+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd813a810 con 0x7f3bd8070570 2024-03-25T19:38:30.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.384+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184188662 0 0) 0x7f3bd81dad30 con 0x7f3bd80727a0 2024-03-25T19:38:30.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.384+0000 7f3bcdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:30.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.384+0000 7f3bcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:30.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.384+0000 7f3bcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:30.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.384+0000 7f3bcdffb640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:38:30.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.384+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd81db110 con 0x7f3bd80727a0 2024-03-25T19:38:30.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.385+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2842182976 0 0) 0x7f3bbc004eb0 con 0x7f3bd813fc60 2024-03-25T19:38:30.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.385+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bd81dad30 con 0x7f3bd813fc60 2024-03-25T19:38:30.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.385+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3bd0001cd0 con 0x7f3bd813fc60 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.385+0000 7f3bcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.385+0000 7f3bcdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.385+0000 7f3bcdffb640 10 monclient(hunting): dump: 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:30.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.385+0000 7f3bcdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:30.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.399+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3455311707 0 0) 0x7f3bd813a810 con 0x7f3bd8070570 2024-03-25T19:38:30.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.399+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bbc004eb0 con 0x7f3bd8070570 2024-03-25T19:38:30.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.399+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3bd4001670 con 0x7f3bd8070570 2024-03-25T19:38:30.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.399+0000 7f3bcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.399+0000 7f3bcdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:30.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.399+0000 7f3bcdffb640 10 monclient(hunting): dump: 2024-03-25T19:38:30.399 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:30.399 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:30.399 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:30.399 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:30.399 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.399 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:30.400 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:30.400 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:30.400 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:30.400 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.404+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 282878537 0 0) 0x7f3bd81db110 con 0x7f3bd80727a0 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.404+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bd813a810 con 0x7f3bd80727a0 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.404+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3bc8001830 con 0x7f3bd80727a0 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.404+0000 7f3bcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.404+0000 7f3bcdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.404+0000 7f3bcdffb640 10 monclient(hunting): dump: 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.404 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:30.405 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:30.405 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:30.405 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:30.405 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:30.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1668310994 0 0) 0x7f3bd81dad30 con 0x7f3bd813fc60 2024-03-25T19:38:30.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 >> v1:172.21.15.19:6790/0 conn(0x7f3bd80727a0 legacy=0x7f3bd81db630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 >> v1:172.21.15.19:6789/0 conn(0x7f3bd8070570 legacy=0x7f3bd81d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bcdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:30.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bcdffb640 1 monclient: found mon.b 2024-03-25T19:38:30.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:30.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bcdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:30.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd81daec0 con 0x7f3bd813fc60 2024-03-25T19:38:30.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bcdffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:30.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bcdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:30.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:30.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bdf843640 5 monclient: authenticate success, global_id 4648 2024-03-25T19:38:30.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bdf843640 10 monclient: _renew_subs 2024-03-25T19:38:30.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bdf843640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:30.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bdf843640 1 -- 172.21.15.19:0/2878114384 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd8136a20 con 0x7f3bd813fc60 2024-03-25T19:38:30.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bdf843640 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:30.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bdf843640 10 monclient: _renew_subs 2024-03-25T19:38:30.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bdf843640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:30.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bdf843640 1 -- 172.21.15.19:0/2878114384 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd8139c60 con 0x7f3bd813fc60 2024-03-25T19:38:30.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.405+0000 7f3bdf843640 1 librados: init done 2024-03-25T19:38:30.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.407+0000 7f3bdf843640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:30.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.407+0000 7f3bdf843640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:30.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.407+0000 7f3bdf843640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:30.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.407+0000 7f3bdf843640 1 -- 172.21.15.19:0/2878114384 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd80627b0 con 0x7f3bd813fc60 2024-03-25T19:38:30.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.692+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bd00020e0 con 0x7f3bd813fc60 2024-03-25T19:38:30.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.692+0000 7f3bcdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:30.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.692+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f3bd0003b30 con 0x7f3bd813fc60 2024-03-25T19:38:30.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.692+0000 7f3bcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.692+0000 7f3bcdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.692+0000 7f3bcdffb640 10 monclient: dump: 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:30.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.692+0000 7f3bcf7fe640 4 set_mon_vals no callback set 2024-03-25T19:38:30.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.694+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f3bd0019b40 con 0x7f3bd813fc60 2024-03-25T19:38:30.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.694+0000 7f3bdd5df640 10 monclient: get_auth_request con 0x7f3bbc06e0f0 auth_method 0 2024-03-25T19:38:30.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.695+0000 7f3bcdffb640 10 client.4648.objecter ms_handle_connect 0x7f3bbc06e0f0 2024-03-25T19:38:30.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.698+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f3bd00537e0 con 0x7f3bd813fc60 2024-03-25T19:38:30.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.698+0000 7f3bcdffb640 10 client.4648.objecter ms_dispatch 0x7f3bd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:30.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.698+0000 7f3bcdffb640 3 client.4648.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:30.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.698+0000 7f3bcdffb640 3 client.4648.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:30.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.698+0000 7f3bcdffb640 20 client.4648.objecter dump_active .. 0 homeless 2024-03-25T19:38:30.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.699+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3bd004f2f0 con 0x7f3bd813fc60 2024-03-25T19:38:30.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.699+0000 7f3bcdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:30.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.699+0000 7f3bcdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:30.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.966+0000 7f3bdf843640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:30.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.966+0000 7f3bdf843640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:30.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.966+0000 7f3bdf843640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:30.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.966+0000 7f3bdf843640 1 -- 172.21.15.19:0/2878114384 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bd806b410 con 0x7f3bd813fc60 2024-03-25T19:38:30.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.994+0000 7f3bcdffb640 1 -- 172.21.15.19:0/2878114384 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f3bd0051220 con 0x7f3bd813fc60 2024-03-25T19:38:30.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.994+0000 7f3bcdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:30.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.994+0000 7f3bcdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:30.995 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:38:30.995 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:30.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 10 librados: watch_flush enter 2024-03-25T19:38:30.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 10 librados: watch_flush exit 2024-03-25T19:38:30.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 20 client.4648.objecter shutdown clearing up homeless session... 2024-03-25T19:38:30.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 10 client.4648.objecter successfully canceled tick 2024-03-25T19:38:30.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 1 -- 172.21.15.19:0/2878114384 >> v1:172.21.15.188:6800/2207506229 conn(0x7f3bbc06e0f0 legacy=0x7f3bbc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 10 monclient: shutdown 2024-03-25T19:38:30.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:30.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 1 -- 172.21.15.19:0/2878114384 >> v1:172.21.15.188:6789/0 conn(0x7f3bd813fc60 legacy=0x7f3bd81ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 1 -- 172.21.15.19:0/2878114384 shutdown_connections 2024-03-25T19:38:30.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 1 -- 172.21.15.19:0/2878114384 >> 172.21.15.19:0/2878114384 conn(0x7f3bd80b4a80 msgr2=0x7f3bd80afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:30.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 1 -- 172.21.15.19:0/2878114384 shutdown_connections 2024-03-25T19:38:30.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 1 -- 172.21.15.19:0/2878114384 wait complete. 2024-03-25T19:38:30.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:30.997+0000 7f3bdf843640 1 librados: shutdown 2024-03-25T19:38:31.010 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:37.817 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:37.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.932+0000 7f906fa8d640 10 monclient: get_monmap_and_config 2024-03-25T19:38:37.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.932+0000 7f906fa8d640 10 monclient: build_initial_monmap 2024-03-25T19:38:37.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.932+0000 7f906fa8d640 10 monclient: monmap: 2024-03-25T19:38:37.932 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:37.932 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:37.932 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:38:37.933382+0000 2024-03-25T19:38:37.932 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:38:37.933382+0000 2024-03-25T19:38:37.932 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:37.932 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:37.933 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:37.933 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:38:37.933 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:38:37.933 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:37.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.932+0000 7f906fa8d640 1 Processor -- start 2024-03-25T19:38:37.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.933+0000 7f906fa8d640 1 -- start start 2024-03-25T19:38:37.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.933+0000 7f906fa8d640 10 monclient: init 2024-03-25T19:38:37.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.933+0000 7f906fa8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:37.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.933+0000 7f906fa8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:37.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.933+0000 7f906fa8d640 10 monclient(hunting): picked mon.noname-a con 0x7f90680a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:37.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.933+0000 7f906fa8d640 10 monclient(hunting): picked mon.noname-c con 0x7f906813f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:37.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.933+0000 7f906fa8d640 10 monclient(hunting): picked mon.noname-b con 0x7f90681430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:37.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.933+0000 7f906fa8d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9068142520 con 0x7f90680a98f0 2024-03-25T19:38:37.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.933+0000 7f906fa8d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90680b28a0 con 0x7f906813f710 2024-03-25T19:38:37.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.933+0000 7f906fa8d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90680b2a30 con 0x7f90681430c0 2024-03-25T19:38:37.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.933+0000 7f906fa8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:37.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.933+0000 7f906fa8d640 10 monclient(hunting): authenticate will time out at 6571.203890s 2024-03-25T19:38:37.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.944+0000 7f906e02a640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f90681430c0 0x7f906814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:48986/0 (socket says 172.21.15.19:48986) 2024-03-25T19:38:37.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.944+0000 7f906e02a640 1 -- 172.21.15.19:0/4004084430 learned_addr learned my addr 172.21.15.19:0/4004084430 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:37.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.955+0000 7f906c827640 1 -- 172.21.15.19:0/4004084430 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068499152 0 0) 0x7f90680b28a0 con 0x7f906813f710 2024-03-25T19:38:37.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.955+0000 7f906c827640 10 monclient(hunting): _negotiate 2024-03-25T19:38:37.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.955+0000 7f906c827640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:37.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.955+0000 7f906c827640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:37.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.955+0000 7f906c827640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:38:37.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.955+0000 7f906c827640 1 -- 172.21.15.19:0/4004084430 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9054003520 con 0x7f906813f710 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.960+0000 7f906c827640 1 -- 172.21.15.19:0/4004084430 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585484731 0 0) 0x7f90680b2a30 con 0x7f90681430c0 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.960+0000 7f906c827640 10 monclient(hunting): _negotiate 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.960+0000 7f906c827640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.960+0000 7f906c827640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.960+0000 7f906c827640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.960+0000 7f906c827640 1 -- 172.21.15.19:0/4004084430 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90680b28a0 con 0x7f90681430c0 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.960+0000 7f906c827640 1 -- 172.21.15.19:0/4004084430 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 301357743 0 0) 0x7f9054003520 con 0x7f906813f710 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.960+0000 7f906c827640 1 -- 172.21.15.19:0/4004084430 >> v1:172.21.15.19:6790/0 conn(0x7f90681430c0 legacy=0x7f906814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.960+0000 7f906c827640 1 -- 172.21.15.19:0/4004084430 >> v1:172.21.15.19:6789/0 conn(0x7f90680a98f0 legacy=0x7f90680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.960+0000 7f906c827640 10 monclient: _finish_hunting 0 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.960+0000 7f906c827640 1 monclient: found mon.noname-c 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.960+0000 7f906c827640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.961+0000 7f906c827640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:38:37.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.961+0000 7f906c827640 1 -- 172.21.15.19:0/4004084430 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90680b44c0 con 0x7f906813f710 2024-03-25T19:38:37.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.961+0000 7f906c827640 10 monclient: _finish_auth 0 2024-03-25T19:38:37.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.961+0000 7f906c827640 10 monclient: _check_auth_tickets 2024-03-25T19:38:37.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.961+0000 7f906c827640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:37.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.961+0000 7f906fa8d640 5 monclient: authenticate success, global_id 4651 2024-03-25T19:38:37.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.961+0000 7f906fa8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:37.963 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.963+0000 7f906c827640 1 -- 172.21.15.19:0/4004084430 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9058001650 con 0x7f906813f710 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.963+0000 7f906c827640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.963+0000 7f906c827640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.963+0000 7f906c827640 10 monclient: dump: 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:37.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.963+0000 7f906fa8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:37.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.986+0000 7f906c827640 1 -- 172.21.15.19:0/4004084430 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90580018a0 con 0x7f906813f710 2024-03-25T19:38:37.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.986+0000 7f906c827640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:37.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.986+0000 7f906c827640 1 -- 172.21.15.19:0/4004084430 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9058001c30 con 0x7f906813f710 2024-03-25T19:38:37.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.986+0000 7f906c827640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:37.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.986+0000 7f906c827640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:37.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.986+0000 7f906c827640 10 monclient: dump: 2024-03-25T19:38:37.986 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:37.986 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:37.986 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:37.986 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:37.987 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:37.987 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:37.987 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:37.987 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:37.987 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:37.987 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:37.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.986+0000 7f906fa8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:37.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.986+0000 7f906fa8d640 4 set_mon_vals no callback set 2024-03-25T19:38:37.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.986+0000 7f906fa8d640 10 monclient: shutdown 2024-03-25T19:38:37.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.986+0000 7f906fa8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.986+0000 7f906fa8d640 1 -- 172.21.15.19:0/4004084430 >> v1:172.21.15.188:6789/0 conn(0x7f906813f710 legacy=0x7f90681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.987+0000 7f906fa8d640 1 -- 172.21.15.19:0/4004084430 shutdown_connections 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.987+0000 7f906fa8d640 1 -- 172.21.15.19:0/4004084430 >> 172.21.15.19:0/4004084430 conn(0x7f90680b7ea0 msgr2=0x7f90680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.987+0000 7f906fa8d640 1 -- 172.21.15.19:0/4004084430 shutdown_connections 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.987+0000 7f906fa8d640 1 -- 172.21.15.19:0/4004084430 wait complete. 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.988+0000 7f906fa8d640 10 monclient: build_initial_monmap 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.988+0000 7f906fa8d640 10 monclient: monmap: 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:38:37.989 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:37.990 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:38:37.990 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:38:37.990 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:37.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.988+0000 7f906fa8d640 1 librados: starting msgr at 2024-03-25T19:38:37.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.988+0000 7f906fa8d640 1 librados: starting objecter 2024-03-25T19:38:37.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.988+0000 7f906fa8d640 1 Processor -- start 2024-03-25T19:38:37.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.988+0000 7f906fa8d640 1 -- start start 2024-03-25T19:38:37.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.988+0000 7f906fa8d640 1 librados: setting wanted keys 2024-03-25T19:38:37.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.988+0000 7f906fa8d640 1 librados: calling monclient init 2024-03-25T19:38:37.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.988+0000 7f906fa8d640 10 monclient: init 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.989+0000 7f906fa8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.989+0000 7f906fa8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.989+0000 7f906fa8d640 10 monclient(hunting): picked mon.noname-b con 0x7f90680a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.989+0000 7f906fa8d640 10 monclient(hunting): picked mon.noname-a con 0x7f906813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.989+0000 7f906fa8d640 10 monclient(hunting): picked mon.noname-c con 0x7f90681430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.989+0000 7f906fa8d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9068139180 con 0x7f906813f710 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.989+0000 7f906fa8d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90680abb30 con 0x7f90680a98f0 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.989+0000 7f906fa8d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90680abda0 con 0x7f90681430c0 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.989+0000 7f906fa8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.989+0000 7f906fa8d640 10 monclient(hunting): authenticate will time out at 6571.259527s 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.991+0000 7f906e02a640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f90681430c0 0x7f90681e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:49008/0 (socket says 172.21.15.19:49008) 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.991+0000 7f906e02a640 1 -- 172.21.15.19:0/3743006923 learned_addr learned my addr 172.21.15.19:0/3743006923 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.992+0000 7f905e7fc640 10 client.?.objecter ms_handle_connect 0x7f90681430c0 2024-03-25T19:38:37.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.992+0000 7f905e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:37.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.998+0000 7f905e7fc640 10 client.?.objecter ms_handle_connect 0x7f906813f710 2024-03-25T19:38:37.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.998+0000 7f905e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:37.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.998+0000 7f905e7fc640 10 client.?.objecter ms_handle_connect 0x7f90680a98f0 2024-03-25T19:38:37.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:37.998+0000 7f905e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:38.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.074+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947684770 0 0) 0x7f90680abda0 con 0x7f90681430c0 2024-03-25T19:38:38.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.074+0000 7f905e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:38.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.074+0000 7f905e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:38.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.074+0000 7f905e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:38.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.074+0000 7f905e7fc640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:38:38.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.074+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904c004eb0 con 0x7f90681430c0 2024-03-25T19:38:38.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.076+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420905257 0 0) 0x7f9068139180 con 0x7f906813f710 2024-03-25T19:38:38.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.076+0000 7f905e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:38.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.076+0000 7f905e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:38.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.076+0000 7f905e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:38.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.076+0000 7f905e7fc640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:38:38.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.076+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90680abda0 con 0x7f906813f710 2024-03-25T19:38:38.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.091+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1041841069 0 0) 0x7f904c004eb0 con 0x7f90681430c0 2024-03-25T19:38:38.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.091+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9068139180 con 0x7f90681430c0 2024-03-25T19:38:38.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.091+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9060001c10 con 0x7f90681430c0 2024-03-25T19:38:38.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.091+0000 7f905e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:38.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.091+0000 7f905e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:38.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.091+0000 7f905e7fc640 10 monclient(hunting): dump: 2024-03-25T19:38:38.091 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:38.091 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:38.092 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:38.092 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:38.092 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:38.092 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:38.092 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:38.092 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:38.092 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:38.092 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:38.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.091+0000 7f905e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:38.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.093+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2917540463 0 0) 0x7f90680abda0 con 0x7f906813f710 2024-03-25T19:38:38.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.093+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f904c004eb0 con 0x7f906813f710 2024-03-25T19:38:38.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.093+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9058001bb0 con 0x7f906813f710 2024-03-25T19:38:38.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.093+0000 7f905e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:38.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.093+0000 7f905e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:38.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.093+0000 7f905e7fc640 10 monclient(hunting): dump: 2024-03-25T19:38:38.093 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:38.093 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:38.093 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:38.093 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:38.094 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:38.094 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:38.094 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:38.094 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:38.094 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:38.094 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:38.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1987867151 0 0) 0x7f9068139180 con 0x7f90681430c0 2024-03-25T19:38:38.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 >> v1:172.21.15.188:6789/0 conn(0x7f90680a98f0 legacy=0x7f90680ac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 >> v1:172.21.15.19:6789/0 conn(0x7f906813f710 legacy=0x7f9068136580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f905e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:38.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f905e7fc640 1 monclient: found mon.c 2024-03-25T19:38:38.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f905e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:38.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f905e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:38:38.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90680ac080 con 0x7f90681430c0 2024-03-25T19:38:38.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f905e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:38.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f905e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:38.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f905e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:38.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f906fa8d640 5 monclient: authenticate success, global_id 4628 2024-03-25T19:38:38.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f906fa8d640 10 monclient: _renew_subs 2024-03-25T19:38:38.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f906fa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:38:38.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f906fa8d640 1 -- 172.21.15.19:0/3743006923 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9068137d90 con 0x7f90681430c0 2024-03-25T19:38:38.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f906fa8d640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:38.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f906fa8d640 10 monclient: _renew_subs 2024-03-25T19:38:38.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f906fa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:38:38.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f906fa8d640 1 -- 172.21.15.19:0/3743006923 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9068138b60 con 0x7f90681430c0 2024-03-25T19:38:38.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.100+0000 7f906fa8d640 1 librados: init done 2024-03-25T19:38:38.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.101+0000 7f906fa8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:38.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.101+0000 7f906fa8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:38.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.101+0000 7f906fa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:38:38.103 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.101+0000 7f906fa8d640 1 -- 172.21.15.19:0/3743006923 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f906814f2d0 con 0x7f90681430c0 2024-03-25T19:38:38.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.110+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9060004b60 con 0x7f90681430c0 2024-03-25T19:38:38.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.110+0000 7f905e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:38.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.110+0000 7f905ffff640 4 set_mon_vals no callback set 2024-03-25T19:38:38.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.411+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f9060004e70 con 0x7f90681430c0 2024-03-25T19:38:38.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.411+0000 7f905e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:38.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.411+0000 7f905e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:38.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.411+0000 7f905e7fc640 10 monclient: dump: 2024-03-25T19:38:38.411 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:38.411 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:38.411 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:38.411 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:38.411 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:38.411 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:38.412 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:38.412 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:38.412 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:38.412 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:38.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.411+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f906001ae40 con 0x7f90681430c0 2024-03-25T19:38:38.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.413+0000 7f906d829640 10 monclient: get_auth_request con 0x7f904c06de40 auth_method 0 2024-03-25T19:38:38.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.413+0000 7f905e7fc640 10 client.4628.objecter ms_handle_connect 0x7f904c06de40 2024-03-25T19:38:38.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.422+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f9060054b50 con 0x7f90681430c0 2024-03-25T19:38:38.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.422+0000 7f905e7fc640 10 client.4628.objecter ms_dispatch 0x7f9068000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:38.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.422+0000 7f905e7fc640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:38.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.422+0000 7f905e7fc640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:38.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.422+0000 7f905e7fc640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T19:38:38.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.422+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9060050490 con 0x7f90681430c0 2024-03-25T19:38:38.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.422+0000 7f905e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:38.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.422+0000 7f905e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:38.443 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:01.172 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.499 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:01.228 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.529 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:01.258 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.551 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:10:01.254 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.557 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:10:01.260 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.561 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:01.291 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.579 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:01.308 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.596 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:10:01.299 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.609 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:01.338 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.669+0000 7f906fa8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:38.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.669+0000 7f906fa8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:38.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.669+0000 7f906fa8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:38:38.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.669+0000 7f906fa8d640 1 -- 172.21.15.19:0/3743006923 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9068139330 con 0x7f90681430c0 2024-03-25T19:38:38.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.735+0000 7f905e7fc640 1 -- 172.21.15.19:0/3743006923 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f9060052590 con 0x7f90681430c0 2024-03-25T19:38:38.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.735+0000 7f905e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:38.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.735+0000 7f905e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:38.734 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:38:38.734 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:38.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 10 librados: watch_flush enter 2024-03-25T19:38:38.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 10 librados: watch_flush exit 2024-03-25T19:38:38.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T19:38:38.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 10 client.4628.objecter successfully canceled tick 2024-03-25T19:38:38.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 1 -- 172.21.15.19:0/3743006923 >> v1:172.21.15.188:6800/2207506229 conn(0x7f904c06de40 legacy=0x7f904c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 10 monclient: shutdown 2024-03-25T19:38:38.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:38.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 1 -- 172.21.15.19:0/3743006923 >> v1:172.21.15.19:6790/0 conn(0x7f90681430c0 legacy=0x7f90681e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 1 -- 172.21.15.19:0/3743006923 shutdown_connections 2024-03-25T19:38:38.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 1 -- 172.21.15.19:0/3743006923 >> 172.21.15.19:0/3743006923 conn(0x7f90680b7ea0 msgr2=0x7f9068074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:38.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 1 -- 172.21.15.19:0/3743006923 shutdown_connections 2024-03-25T19:38:38.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 1 -- 172.21.15.19:0/3743006923 wait complete. 2024-03-25T19:38:38.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:38.737+0000 7f906fa8d640 1 librados: shutdown 2024-03-25T19:38:38.747 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:45.554 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:45.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.670+0000 7fcc30dc2640 10 monclient: get_monmap_and_config 2024-03-25T19:38:45.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.670+0000 7fcc30dc2640 10 monclient: build_initial_monmap 2024-03-25T19:38:45.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.670+0000 7fcc30dc2640 10 monclient: monmap: 2024-03-25T19:38:45.669 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:45.669 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:45.669 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:38:45.670450+0000 2024-03-25T19:38:45.669 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:38:45.670450+0000 2024-03-25T19:38:45.669 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:45.669 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:45.669 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:45.670 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:38:45.670 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:38:45.670 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:45.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.671+0000 7fcc30dc2640 1 Processor -- start 2024-03-25T19:38:45.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.671+0000 7fcc30dc2640 1 -- start start 2024-03-25T19:38:45.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.671+0000 7fcc30dc2640 10 monclient: init 2024-03-25T19:38:45.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.672+0000 7fcc30dc2640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:45.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.672+0000 7fcc30dc2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.672+0000 7fcc30dc2640 10 monclient(hunting): picked mon.noname-b con 0x7fcc2c13fc60 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.672+0000 7fcc30dc2640 10 monclient(hunting): picked mon.noname-c con 0x7fcc2c070570 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.672+0000 7fcc30dc2640 10 monclient(hunting): picked mon.noname-a con 0x7fcc2c0727a0 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.672+0000 7fcc30dc2640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c13c9f0 con 0x7fcc2c0727a0 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.672+0000 7fcc30dc2640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c06fac0 con 0x7fcc2c070570 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.672+0000 7fcc30dc2640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c0715d0 con 0x7fcc2c13fc60 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.672+0000 7fcc30dc2640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:45.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.672+0000 7fcc30dc2640 10 monclient(hunting): authenticate will time out at 6578.941448s 2024-03-25T19:38:45.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.677+0000 7fcc2a59c640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fcc2c13fc60 0x7fcc2c06f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:36166/0 (socket says 172.21.15.19:36166) 2024-03-25T19:38:45.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.677+0000 7fcc2a59c640 1 -- 172.21.15.19:0/4194507288 learned_addr learned my addr 172.21.15.19:0/4194507288 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:45.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.685+0000 7fcc2959a640 1 -- 172.21.15.19:0/4194507288 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951452101 0 0) 0x7fcc2c06fac0 con 0x7fcc2c070570 2024-03-25T19:38:45.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.685+0000 7fcc2959a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.685+0000 7fcc2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.685+0000 7fcc2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.685+0000 7fcc2959a640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:38:45.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.685+0000 7fcc2959a640 1 -- 172.21.15.19:0/4194507288 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc14003520 con 0x7fcc2c070570 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.689+0000 7fcc2959a640 1 -- 172.21.15.19:0/4194507288 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251490782 0 0) 0x7fcc2c0715d0 con 0x7fcc2c13fc60 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.689+0000 7fcc2959a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.689+0000 7fcc2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.689+0000 7fcc2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.689+0000 7fcc2959a640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:38:45.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.689+0000 7fcc2959a640 1 -- 172.21.15.19:0/4194507288 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2c06fac0 con 0x7fcc2c13fc60 2024-03-25T19:38:45.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 1 -- 172.21.15.19:0/4194507288 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2592028019 0 0) 0x7fcc14003520 con 0x7fcc2c070570 2024-03-25T19:38:45.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 1 -- 172.21.15.19:0/4194507288 >> v1:172.21.15.19:6790/0 conn(0x7fcc2c13fc60 legacy=0x7fcc2c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 1 -- 172.21.15.19:0/4194507288 >> v1:172.21.15.19:6789/0 conn(0x7fcc2c0727a0 legacy=0x7fcc2c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 10 monclient: _finish_hunting 0 2024-03-25T19:38:45.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 1 monclient: found mon.noname-c 2024-03-25T19:38:45.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:45.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:38:45.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 1 -- 172.21.15.19:0/4194507288 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c0718b0 con 0x7fcc2c070570 2024-03-25T19:38:45.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 10 monclient: _finish_auth 0 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 10 monclient: _check_auth_tickets 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 1 -- 172.21.15.19:0/4194507288 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fcc20001690 con 0x7fcc2c070570 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc2959a640 10 monclient: dump: 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:45.700 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:45.701 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:45.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc30dc2640 5 monclient: authenticate success, global_id 4654 2024-03-25T19:38:45.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.700+0000 7fcc30dc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:45.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.705+0000 7fcc2959a640 1 -- 172.21.15.19:0/4194507288 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc200017f0 con 0x7fcc2c070570 2024-03-25T19:38:45.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.705+0000 7fcc2959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:45.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.705+0000 7fcc2959a640 1 -- 172.21.15.19:0/4194507288 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fcc20001c00 con 0x7fcc2c070570 2024-03-25T19:38:45.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.705+0000 7fcc30dc2640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:45.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.705+0000 7fcc30dc2640 4 set_mon_vals no callback set 2024-03-25T19:38:45.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.705+0000 7fcc2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.705+0000 7fcc2959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:45.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.705+0000 7fcc2959a640 10 monclient: dump: 2024-03-25T19:38:45.705 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:45.705 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:45.705 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:45.705 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:45.705 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.705 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:45.705 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:45.705 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:45.705 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:45.705 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:45.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.706+0000 7fcc30dc2640 10 monclient: shutdown 2024-03-25T19:38:45.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.706+0000 7fcc30dc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:45.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.706+0000 7fcc30dc2640 1 -- 172.21.15.19:0/4194507288 >> v1:172.21.15.188:6789/0 conn(0x7fcc2c070570 legacy=0x7fcc2c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.706+0000 7fcc30dc2640 1 -- 172.21.15.19:0/4194507288 shutdown_connections 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.706+0000 7fcc30dc2640 1 -- 172.21.15.19:0/4194507288 >> 172.21.15.19:0/4194507288 conn(0x7fcc2c0b4a80 msgr2=0x7fcc2c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.706+0000 7fcc30dc2640 1 -- 172.21.15.19:0/4194507288 shutdown_connections 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.706+0000 7fcc30dc2640 1 -- 172.21.15.19:0/4194507288 wait complete. 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.707+0000 7fcc30dc2640 10 monclient: build_initial_monmap 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.707+0000 7fcc30dc2640 10 monclient: monmap: 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:38:45.708 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.707+0000 7fcc30dc2640 1 librados: starting msgr at 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.707+0000 7fcc30dc2640 1 librados: starting objecter 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.707+0000 7fcc30dc2640 1 Processor -- start 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.707+0000 7fcc30dc2640 1 -- start start 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.707+0000 7fcc30dc2640 1 librados: setting wanted keys 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.707+0000 7fcc30dc2640 1 librados: calling monclient init 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.707+0000 7fcc30dc2640 10 monclient: init 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.708+0000 7fcc30dc2640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.708+0000 7fcc30dc2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.708+0000 7fcc30dc2640 10 monclient(hunting): picked mon.noname-c con 0x7fcc2c070570 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.708+0000 7fcc30dc2640 10 monclient(hunting): picked mon.noname-a con 0x7fcc2c0727a0 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.708+0000 7fcc30dc2640 10 monclient(hunting): picked mon.noname-b con 0x7fcc2c13fc60 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.708+0000 7fcc30dc2640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c0ac910 con 0x7fcc2c0727a0 2024-03-25T19:38:45.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.708+0000 7fcc30dc2640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c1d5670 con 0x7fcc2c13fc60 2024-03-25T19:38:45.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.708+0000 7fcc30dc2640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c0ab4f0 con 0x7fcc2c070570 2024-03-25T19:38:45.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.708+0000 7fcc30dc2640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:45.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.708+0000 7fcc30dc2640 10 monclient(hunting): authenticate will time out at 6578.977337s 2024-03-25T19:38:45.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.719+0000 7fcc29d9b640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fcc2c0727a0 0x7fcc2c1d5b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:44250/0 (socket says 172.21.15.19:44250) 2024-03-25T19:38:45.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.719+0000 7fcc29d9b640 1 -- 172.21.15.19:0/1045816992 learned_addr learned my addr 172.21.15.19:0/1045816992 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:45.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.721+0000 7fcc1affd640 10 client.?.objecter ms_handle_connect 0x7fcc2c0727a0 2024-03-25T19:38:45.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.721+0000 7fcc1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:45.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.724+0000 7fcc1affd640 10 client.?.objecter ms_handle_connect 0x7fcc2c13fc60 2024-03-25T19:38:45.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.724+0000 7fcc1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:45.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.725+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001540921 0 0) 0x7fcc2c0ac910 con 0x7fcc2c0727a0 2024-03-25T19:38:45.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.725+0000 7fcc1affd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.725+0000 7fcc1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.725+0000 7fcc1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.725+0000 7fcc1affd640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:38:45.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.725+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0c004eb0 con 0x7fcc2c0727a0 2024-03-25T19:38:45.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.725+0000 7fcc1affd640 10 client.?.objecter ms_handle_connect 0x7fcc2c070570 2024-03-25T19:38:45.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.726+0000 7fcc1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:45.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.731+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901353209 0 0) 0x7fcc2c1d5670 con 0x7fcc2c13fc60 2024-03-25T19:38:45.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.731+0000 7fcc1affd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.731+0000 7fcc1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.731+0000 7fcc1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.731+0000 7fcc1affd640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:38:45.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.731+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2c0ac910 con 0x7fcc2c13fc60 2024-03-25T19:38:45.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.748+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1918306402 0 0) 0x7fcc0c004eb0 con 0x7fcc2c0727a0 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.748+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc2c1d5670 con 0x7fcc2c0727a0 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.748+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fcc20001c00 con 0x7fcc2c0727a0 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.748+0000 7fcc1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.748+0000 7fcc1affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.748+0000 7fcc1affd640 10 monclient(hunting): dump: 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:45.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.748+0000 7fcc1affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:45.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.752+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3859569218 0 0) 0x7fcc2c0ac910 con 0x7fcc2c13fc60 2024-03-25T19:38:45.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.752+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc0c004eb0 con 0x7fcc2c13fc60 2024-03-25T19:38:45.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.752+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fcc24001b10 con 0x7fcc2c13fc60 2024-03-25T19:38:45.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.752+0000 7fcc1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.752+0000 7fcc1affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:45.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.752+0000 7fcc1affd640 10 monclient(hunting): dump: 2024-03-25T19:38:45.751 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:45.751 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:45.751 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:45.751 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:45.752 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.752 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:45.752 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:45.752 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:45.752 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:45.752 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:45.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.754+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581084825 0 0) 0x7fcc2c0ab4f0 con 0x7fcc2c070570 2024-03-25T19:38:45.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.754+0000 7fcc1affd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.754+0000 7fcc1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.754+0000 7fcc1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.754+0000 7fcc1affd640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:38:45.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.754+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2c0ac910 con 0x7fcc2c070570 2024-03-25T19:38:45.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.761+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4174658824 0 0) 0x7fcc2c0ac910 con 0x7fcc2c070570 2024-03-25T19:38:45.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.761+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc2c0ab4f0 con 0x7fcc2c070570 2024-03-25T19:38:45.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.761+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fcc1c0018f0 con 0x7fcc2c070570 2024-03-25T19:38:45.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.761+0000 7fcc1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.761+0000 7fcc1affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:45.760 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.761+0000 7fcc1affd640 10 monclient(hunting): dump: 2024-03-25T19:38:45.760 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:45.761 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:45.761 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:45.761 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:45.761 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.761 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:45.761 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:45.761 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:45.761 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:45.761 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:45.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 131610080 0 0) 0x7fcc2c1d5670 con 0x7fcc2c0727a0 2024-03-25T19:38:45.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 >> v1:172.21.15.19:6790/0 conn(0x7fcc2c070570 legacy=0x7fcc2c0aade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 >> v1:172.21.15.188:6789/0 conn(0x7fcc2c13fc60 legacy=0x7fcc2c136b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc1affd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:45.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc1affd640 1 monclient: found mon.a 2024-03-25T19:38:45.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:45.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc1affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:38:45.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c1d62a0 con 0x7fcc2c0727a0 2024-03-25T19:38:45.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc1affd640 10 monclient: _finish_auth 0 2024-03-25T19:38:45.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc1affd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:45.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:45.767 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc30dc2640 5 monclient: authenticate success, global_id 4662 2024-03-25T19:38:45.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc30dc2640 10 monclient: _renew_subs 2024-03-25T19:38:45.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc30dc2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:38:45.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc30dc2640 1 -- 172.21.15.19:0/1045816992 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2c0aa620 con 0x7fcc2c0727a0 2024-03-25T19:38:45.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc30dc2640 10 client.4662.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:45.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc30dc2640 10 monclient: _renew_subs 2024-03-25T19:38:45.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc30dc2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:38:45.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc30dc2640 1 -- 172.21.15.19:0/1045816992 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2c0ad180 con 0x7fcc2c0727a0 2024-03-25T19:38:45.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.767+0000 7fcc30dc2640 1 librados: init done 2024-03-25T19:38:45.769 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.768+0000 7fcc30dc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:45.769 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.768+0000 7fcc30dc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:45.769 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.768+0000 7fcc30dc2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:38:45.769 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:45.768+0000 7fcc30dc2640 1 -- 172.21.15.19:0/1045816992 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2c1d5820 con 0x7fcc2c0727a0 2024-03-25T19:38:46.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.119+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc20001210 con 0x7fcc2c0727a0 2024-03-25T19:38:46.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.119+0000 7fcc1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:46.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.119+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fcc200038a0 con 0x7fcc2c0727a0 2024-03-25T19:38:46.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.119+0000 7fcc1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:46.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.119+0000 7fcc1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:46.119 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.119+0000 7fcc1affd640 10 monclient: dump: 2024-03-25T19:38:46.119 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:46.119 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:46.119 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:46.119 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:46.119 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:46.120 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:46.120 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:46.120 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:46.120 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:46.120 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:46.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.119+0000 7fcc28d99640 4 set_mon_vals no callback set 2024-03-25T19:38:46.120 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.120+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fcc20003ac0 con 0x7fcc2c0727a0 2024-03-25T19:38:46.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.121+0000 7fcc2a59c640 10 monclient: get_auth_request con 0x7fcc0c06e130 auth_method 0 2024-03-25T19:38:46.121 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.121+0000 7fcc1affd640 10 client.4662.objecter ms_handle_connect 0x7fcc0c06e130 2024-03-25T19:38:46.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.129+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7fcc20054640 con 0x7fcc2c0727a0 2024-03-25T19:38:46.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.129+0000 7fcc1affd640 10 client.4662.objecter ms_dispatch 0x7fcc2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:46.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.129+0000 7fcc1affd640 3 client.4662.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:46.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.129+0000 7fcc1affd640 3 client.4662.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:46.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.129+0000 7fcc1affd640 20 client.4662.objecter dump_active .. 0 homeless 2024-03-25T19:38:46.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.129+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc20050150 con 0x7fcc2c0727a0 2024-03-25T19:38:46.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.129+0000 7fcc1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:46.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.129+0000 7fcc1affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:46.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.349+0000 7fcc30dc2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:46.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.349+0000 7fcc30dc2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:46.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.349+0000 7fcc30dc2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:38:46.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.349+0000 7fcc30dc2640 1 -- 172.21.15.19:0/1045816992 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc2c1d6430 con 0x7fcc2c0727a0 2024-03-25T19:38:46.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.373+0000 7fcc1affd640 1 -- 172.21.15.19:0/1045816992 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7fcc20052080 con 0x7fcc2c0727a0 2024-03-25T19:38:46.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.373+0000 7fcc1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:46.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.373+0000 7fcc1affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:46.373 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:38:46.373 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:46.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.375+0000 7fcc30dc2640 10 librados: watch_flush enter 2024-03-25T19:38:46.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.375+0000 7fcc30dc2640 10 librados: watch_flush exit 2024-03-25T19:38:46.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.375+0000 7fcc30dc2640 20 client.4662.objecter shutdown clearing up homeless session... 2024-03-25T19:38:46.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.375+0000 7fcc30dc2640 10 client.4662.objecter successfully canceled tick 2024-03-25T19:38:46.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.375+0000 7fcc30dc2640 1 -- 172.21.15.19:0/1045816992 >> v1:172.21.15.188:6800/2207506229 conn(0x7fcc0c06e130 legacy=0x7fcc0c0705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:46.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.375+0000 7fcc30dc2640 10 monclient: shutdown 2024-03-25T19:38:46.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.375+0000 7fcc30dc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:46.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.375+0000 7fcc30dc2640 1 -- 172.21.15.19:0/1045816992 >> v1:172.21.15.19:6789/0 conn(0x7fcc2c0727a0 legacy=0x7fcc2c1d5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:46.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.375+0000 7fcc30dc2640 1 -- 172.21.15.19:0/1045816992 shutdown_connections 2024-03-25T19:38:46.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.375+0000 7fcc30dc2640 1 -- 172.21.15.19:0/1045816992 >> 172.21.15.19:0/1045816992 conn(0x7fcc2c0b4a80 msgr2=0x7fcc2c06b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:46.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.376+0000 7fcc30dc2640 1 -- 172.21.15.19:0/1045816992 shutdown_connections 2024-03-25T19:38:46.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.376+0000 7fcc30dc2640 1 -- 172.21.15.19:0/1045816992 wait complete. 2024-03-25T19:38:46.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:46.376+0000 7fcc30dc2640 1 librados: shutdown 2024-03-25T19:38:46.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:53.193 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:53.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.307+0000 7f1d61e8d640 10 monclient: get_monmap_and_config 2024-03-25T19:38:53.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.307+0000 7f1d61e8d640 10 monclient: build_initial_monmap 2024-03-25T19:38:53.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.307+0000 7f1d61e8d640 10 monclient: monmap: 2024-03-25T19:38:53.306 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:53.306 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:53.306 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:38:53.307386+0000 2024-03-25T19:38:53.306 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:38:53.307386+0000 2024-03-25T19:38:53.306 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:53.306 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:53.307 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:53.307 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:38:53.307 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:38:53.307 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:53.308 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.308+0000 7f1d61e8d640 1 Processor -- start 2024-03-25T19:38:53.308 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.308+0000 7f1d61e8d640 1 -- start start 2024-03-25T19:38:53.308 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.308+0000 7f1d61e8d640 10 monclient: init 2024-03-25T19:38:53.308 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.309+0000 7f1d61e8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:53.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.309+0000 7f1d61e8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:53.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.309+0000 7f1d61e8d640 10 monclient(hunting): picked mon.noname-b con 0x7f1d5c0a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:53.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.309+0000 7f1d61e8d640 10 monclient(hunting): picked mon.noname-a con 0x7f1d5c13f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:53.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.309+0000 7f1d61e8d640 10 monclient(hunting): picked mon.noname-c con 0x7f1d5c1430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:53.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.309+0000 7f1d61e8d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c142520 con 0x7f1d5c13f710 2024-03-25T19:38:53.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.309+0000 7f1d61e8d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0b28a0 con 0x7f1d5c1430c0 2024-03-25T19:38:53.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.309+0000 7f1d61e8d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0b2a30 con 0x7f1d5c0a98f0 2024-03-25T19:38:53.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.309+0000 7f1d61e8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:53.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.309+0000 7f1d61e8d640 10 monclient(hunting): authenticate will time out at 6586.578152s 2024-03-25T19:38:53.316 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.317+0000 7f1d5b7fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f1d5c0a98f0 0x7f1d5c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:51206/0 (socket says 172.21.15.19:51206) 2024-03-25T19:38:53.316 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.317+0000 7f1d5b7fe640 1 -- 172.21.15.19:0/1866051294 learned_addr learned my addr 172.21.15.19:0/1866051294 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:53.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.326+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134992973 0 0) 0x7f1d5c0b2a30 con 0x7f1d5c0a98f0 2024-03-25T19:38:53.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.326+0000 7f1d5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:53.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.326+0000 7f1d5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:53.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.326+0000 7f1d5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:53.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.326+0000 7f1d5a7fc640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:38:53.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.326+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d44003520 con 0x7f1d5c0a98f0 2024-03-25T19:38:53.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.334+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265524597 0 0) 0x7f1d5c142520 con 0x7f1d5c13f710 2024-03-25T19:38:53.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.334+0000 7f1d5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:53.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.334+0000 7f1d5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:53.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.334+0000 7f1d5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:53.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.334+0000 7f1d5a7fc640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:38:53.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.334+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c0b2a30 con 0x7f1d5c13f710 2024-03-25T19:38:53.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.337+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934098878 0 0) 0x7f1d5c0b28a0 con 0x7f1d5c1430c0 2024-03-25T19:38:53.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.337+0000 7f1d5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:53.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.337+0000 7f1d5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:53.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.337+0000 7f1d5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:53.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.337+0000 7f1d5a7fc640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:38:53.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.337+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c142520 con 0x7f1d5c1430c0 2024-03-25T19:38:53.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2272925466 0 0) 0x7f1d44003520 con 0x7f1d5c0a98f0 2024-03-25T19:38:53.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 >> v1:172.21.15.188:6789/0 conn(0x7f1d5c1430c0 legacy=0x7f1d5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 >> v1:172.21.15.19:6789/0 conn(0x7f1d5c13f710 legacy=0x7f1d5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:53.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 1 monclient: found mon.noname-b 2024-03-25T19:38:53.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:53.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:38:53.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c0b44c0 con 0x7f1d5c0a98f0 2024-03-25T19:38:53.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:53.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:53.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:53.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1d4c001690 con 0x7f1d5c0a98f0 2024-03-25T19:38:53.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:53.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:53.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d5a7fc640 10 monclient: dump: 2024-03-25T19:38:53.340 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:53.340 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:53.340 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:53.340 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:53.340 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:53.340 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:53.340 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:53.341 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:53.341 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:53.341 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:53.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d61e8d640 5 monclient: authenticate success, global_id 4637 2024-03-25T19:38:53.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.339+0000 7f1d61e8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:53.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.352+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d4c0017f0 con 0x7f1d5c0a98f0 2024-03-25T19:38:53.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.352+0000 7f1d5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:53.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.352+0000 7f1d5a7fc640 1 -- 172.21.15.19:0/1866051294 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1d4c001c00 con 0x7f1d5c0a98f0 2024-03-25T19:38:53.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.352+0000 7f1d5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:53.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.352+0000 7f1d5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:53.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.352+0000 7f1d5a7fc640 10 monclient: dump: 2024-03-25T19:38:53.351 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:53.352 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:53.352 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:53.352 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:53.352 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:53.352 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:53.352 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:53.352 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:53.352 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:53.352 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:53.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.352+0000 7f1d61e8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:53.352 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.352+0000 7f1d61e8d640 4 set_mon_vals no callback set 2024-03-25T19:38:53.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.353+0000 7f1d61e8d640 10 monclient: shutdown 2024-03-25T19:38:53.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.353+0000 7f1d61e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:53.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.353+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1866051294 >> v1:172.21.15.19:6790/0 conn(0x7f1d5c0a98f0 legacy=0x7f1d5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.353+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1866051294 shutdown_connections 2024-03-25T19:38:53.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.353+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1866051294 >> 172.21.15.19:0/1866051294 conn(0x7f1d5c0b7ea0 msgr2=0x7f1d5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:53.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.353+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1866051294 shutdown_connections 2024-03-25T19:38:53.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.353+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1866051294 wait complete. 2024-03-25T19:38:53.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.353+0000 7f1d61e8d640 10 monclient: build_initial_monmap 2024-03-25T19:38:53.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.353+0000 7f1d61e8d640 10 monclient: monmap: 2024-03-25T19:38:53.355 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:38:53.355 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:53.355 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:38:53.355 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:38:53.355 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:53.355 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:38:53.356 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:38:53.356 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:38:53.356 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:38:53.356 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:53.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 1 librados: starting msgr at 2024-03-25T19:38:53.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 1 librados: starting objecter 2024-03-25T19:38:53.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 1 Processor -- start 2024-03-25T19:38:53.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 1 -- start start 2024-03-25T19:38:53.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 1 librados: setting wanted keys 2024-03-25T19:38:53.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 1 librados: calling monclient init 2024-03-25T19:38:53.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 10 monclient: init 2024-03-25T19:38:53.356 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:53.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:53.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 10 monclient(hunting): picked mon.noname-a con 0x7f1d5c0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:38:53.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 10 monclient(hunting): picked mon.noname-b con 0x7f1d5c13f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:38:53.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 10 monclient(hunting): picked mon.noname-c con 0x7f1d5c1430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:38:53.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c1db7d0 con 0x7f1d5c0a98f0 2024-03-25T19:38:53.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.354+0000 7f1d61e8d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0b1420 con 0x7f1d5c13f710 2024-03-25T19:38:53.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.355+0000 7f1d61e8d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0b15b0 con 0x7f1d5c1430c0 2024-03-25T19:38:53.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.355+0000 7f1d61e8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:53.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.355+0000 7f1d61e8d640 10 monclient(hunting): authenticate will time out at 6586.623932s 2024-03-25T19:38:53.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.361+0000 7f1d5bfff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f1d5c1430c0 0x7f1d5c1d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:51254/0 (socket says 172.21.15.19:51254) 2024-03-25T19:38:53.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.361+0000 7f1d5bfff640 1 -- 172.21.15.19:0/1961391255 learned_addr learned my addr 172.21.15.19:0/1961391255 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:38:53.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.362+0000 7f1d43fff640 10 client.?.objecter ms_handle_connect 0x7f1d5c13f710 2024-03-25T19:38:53.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.362+0000 7f1d43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:53.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.364+0000 7f1d43fff640 10 client.?.objecter ms_handle_connect 0x7f1d5c1430c0 2024-03-25T19:38:53.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.364+0000 7f1d43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:53.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.373+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676242426 0 0) 0x7f1d5c0b1420 con 0x7f1d5c13f710 2024-03-25T19:38:53.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.373+0000 7f1d43fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:53.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.373+0000 7f1d43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:53.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.373+0000 7f1d43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:53.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.373+0000 7f1d43fff640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:38:53.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.373+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d48004eb0 con 0x7f1d5c13f710 2024-03-25T19:38:53.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.374+0000 7f1d43fff640 10 client.?.objecter ms_handle_connect 0x7f1d5c0a98f0 2024-03-25T19:38:53.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.374+0000 7f1d43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:53.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.386+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199097945 0 0) 0x7f1d5c1db7d0 con 0x7f1d5c0a98f0 2024-03-25T19:38:53.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.386+0000 7f1d43fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:53.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.386+0000 7f1d43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:53.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.386+0000 7f1d43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:53.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.386+0000 7f1d43fff640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:38:53.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.386+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c0b1420 con 0x7f1d5c0a98f0 2024-03-25T19:38:53.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.387+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527638375 0 0) 0x7f1d5c0b15b0 con 0x7f1d5c1430c0 2024-03-25T19:38:53.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.387+0000 7f1d43fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:53.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.387+0000 7f1d43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:53.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.387+0000 7f1d43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:53.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.387+0000 7f1d43fff640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:38:53.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.387+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c1db7d0 con 0x7f1d5c1430c0 2024-03-25T19:38:53.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.390+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3791320289 0 0) 0x7f1d48004eb0 con 0x7f1d5c13f710 2024-03-25T19:38:53.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.390+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d5c0b15b0 con 0x7f1d5c13f710 2024-03-25T19:38:53.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.390+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1d50001970 con 0x7f1d5c13f710 2024-03-25T19:38:53.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.390+0000 7f1d43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:53.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.390+0000 7f1d43fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:53.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.390+0000 7f1d43fff640 10 monclient(hunting): dump: 2024-03-25T19:38:53.389 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:53.390 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:53.390 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:53.390 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:53.390 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:53.390 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:53.390 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:53.390 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:53.390 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:53.390 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:53.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.390+0000 7f1d43fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:53.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.403+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 609429482 0 0) 0x7f1d5c0b15b0 con 0x7f1d5c13f710 2024-03-25T19:38:53.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.403+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 >> v1:172.21.15.19:6790/0 conn(0x7f1d5c1430c0 legacy=0x7f1d5c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.403+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 >> v1:172.21.15.19:6789/0 conn(0x7f1d5c0a98f0 legacy=0x7f1d5c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.403+0000 7f1d43fff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:53.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.403+0000 7f1d43fff640 1 monclient: found mon.b 2024-03-25T19:38:53.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.403+0000 7f1d43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.403+0000 7f1d43fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.403+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c0b1740 con 0x7f1d5c13f710 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.404+0000 7f1d43fff640 10 monclient: _finish_auth 0 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.404+0000 7f1d43fff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.404+0000 7f1d43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.404+0000 7f1d61e8d640 5 monclient: authenticate success, global_id 4663 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.404+0000 7f1d61e8d640 10 monclient: _renew_subs 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.404+0000 7f1d61e8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.404+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1961391255 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5c1db1a0 con 0x7f1d5c13f710 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.404+0000 7f1d61e8d640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.404+0000 7f1d61e8d640 10 monclient: _renew_subs 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.404+0000 7f1d61e8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.404+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1961391255 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5c0b08f0 con 0x7f1d5c13f710 2024-03-25T19:38:53.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.404+0000 7f1d61e8d640 1 librados: init done 2024-03-25T19:38:53.405 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.405+0000 7f1d61e8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:53.405 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.405+0000 7f1d61e8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:53.405 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.405+0000 7f1d61e8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:53.405 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.405+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1961391255 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5c0627b0 con 0x7f1d5c13f710 2024-03-25T19:38:53.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.672+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d50003610 con 0x7f1d5c13f710 2024-03-25T19:38:53.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.672+0000 7f1d43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:53.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.672+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1d50003920 con 0x7f1d5c13f710 2024-03-25T19:38:53.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.672+0000 7f1d43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.672+0000 7f1d43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.672+0000 7f1d43fff640 10 monclient: dump: 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:38:53.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.672+0000 7f1d59ffb640 4 set_mon_vals no callback set 2024-03-25T19:38:53.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.673+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f1d50019930 con 0x7f1d5c13f710 2024-03-25T19:38:53.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.674+0000 7f1d5b7fe640 10 monclient: get_auth_request con 0x7f1d4806dda0 auth_method 0 2024-03-25T19:38:53.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.674+0000 7f1d43fff640 10 client.4663.objecter ms_handle_connect 0x7f1d4806dda0 2024-03-25T19:38:53.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.677+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4143411067 0 0) 0x7f1d50053640 con 0x7f1d5c13f710 2024-03-25T19:38:53.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.677+0000 7f1d43fff640 10 client.4663.objecter ms_dispatch 0x7f1d5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:53.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.677+0000 7f1d43fff640 3 client.4663.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:53.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.677+0000 7f1d43fff640 3 client.4663.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:53.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.677+0000 7f1d43fff640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T19:38:53.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.679+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d5004ef80 con 0x7f1d5c13f710 2024-03-25T19:38:53.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.679+0000 7f1d43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:53.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.679+0000 7f1d43fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:53.928 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.929+0000 7f1d61e8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:53.928 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.929+0000 7f1d61e8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:53.928 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.929+0000 7f1d61e8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:38:53.928 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.929+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1961391255 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d5c0abb10 con 0x7f1d5c13f710 2024-03-25T19:38:53.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.954+0000 7f1d43fff640 1 -- 172.21.15.19:0/1961391255 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 682052319) 0x7f1d50051080 con 0x7f1d5c13f710 2024-03-25T19:38:53.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.954+0000 7f1d43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:53.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.954+0000 7f1d43fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:53.953 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:38:53.953 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:29:00.783396+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:29:00.783396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:53.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.956+0000 7f1d61e8d640 10 librados: watch_flush enter 2024-03-25T19:38:53.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.956+0000 7f1d61e8d640 10 librados: watch_flush exit 2024-03-25T19:38:53.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.956+0000 7f1d61e8d640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T19:38:53.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.956+0000 7f1d61e8d640 10 client.4663.objecter successfully canceled tick 2024-03-25T19:38:53.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.956+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1961391255 >> v1:172.21.15.188:6800/2207506229 conn(0x7f1d4806dda0 legacy=0x7f1d48070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.956+0000 7f1d61e8d640 10 monclient: shutdown 2024-03-25T19:38:53.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.956+0000 7f1d61e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:53.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.956+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1961391255 >> v1:172.21.15.188:6789/0 conn(0x7f1d5c13f710 legacy=0x7f1d5c0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.957+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1961391255 shutdown_connections 2024-03-25T19:38:53.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.957+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1961391255 >> 172.21.15.19:0/1961391255 conn(0x7f1d5c0b7ea0 msgr2=0x7f1d5c13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:53.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.957+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1961391255 shutdown_connections 2024-03-25T19:38:53.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.957+0000 7f1d61e8d640 1 -- 172.21.15.19:0/1961391255 wait complete. 2024-03-25T19:38:53.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:38:53.957+0000 7f1d61e8d640 1 librados: shutdown 2024-03-25T19:38:53.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:58.704 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:21.433 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.752 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:21.482 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.780 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:21.510 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.804 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:21.534 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.809 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:21.539 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.814 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:10:21.517 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.825 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:10:21.528 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.832 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:21.562 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.842 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:10:21.545 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:00.775 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:00.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.890+0000 7fc2e34ff640 10 monclient: get_monmap_and_config 2024-03-25T19:39:00.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.890+0000 7fc2e34ff640 10 monclient: build_initial_monmap 2024-03-25T19:39:00.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.890+0000 7fc2e34ff640 10 monclient: monmap: 2024-03-25T19:39:00.890 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:00.890 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:00.890 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:39:00.891351+0000 2024-03-25T19:39:00.890 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:39:00.891351+0000 2024-03-25T19:39:00.890 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:00.890 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:00.890 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:00.890 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:39:00.890 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:39:00.890 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:00.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.890+0000 7fc2e34ff640 1 Processor -- start 2024-03-25T19:39:00.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.891+0000 7fc2e34ff640 1 -- start start 2024-03-25T19:39:00.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.891+0000 7fc2e34ff640 10 monclient: init 2024-03-25T19:39:00.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.891+0000 7fc2e34ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:00.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.891+0000 7fc2e34ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:00.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.891+0000 7fc2e34ff640 10 monclient(hunting): picked mon.noname-a con 0x7fc2dc0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:00.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.891+0000 7fc2e34ff640 10 monclient(hunting): picked mon.noname-b con 0x7fc2dc13f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:00.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.891+0000 7fc2e34ff640 10 monclient(hunting): picked mon.noname-c con 0x7fc2dc1430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:00.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.892+0000 7fc2e34ff640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc142520 con 0x7fc2dc0a98f0 2024-03-25T19:39:00.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.892+0000 7fc2e34ff640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc0b28a0 con 0x7fc2dc1430c0 2024-03-25T19:39:00.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.892+0000 7fc2e34ff640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc0b2a30 con 0x7fc2dc13f710 2024-03-25T19:39:00.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.892+0000 7fc2e34ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:00.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.892+0000 7fc2e34ff640 10 monclient(hunting): authenticate will time out at 6594.162068s 2024-03-25T19:39:00.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.895+0000 7fc2e1a9c640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fc2dc1430c0 0x7fc2dc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:52484/0 (socket says 172.21.15.19:52484) 2024-03-25T19:39:00.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.895+0000 7fc2e1a9c640 1 -- 172.21.15.19:0/2549231053 learned_addr learned my addr 172.21.15.19:0/2549231053 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:00.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.902+0000 7fc2d3fff640 1 -- 172.21.15.19:0/2549231053 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326742835 0 0) 0x7fc2dc0b2a30 con 0x7fc2dc13f710 2024-03-25T19:39:00.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.902+0000 7fc2d3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.902+0000 7fc2d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.902+0000 7fc2d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.902+0000 7fc2d3fff640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:39:00.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.902+0000 7fc2d3fff640 1 -- 172.21.15.19:0/2549231053 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c8003520 con 0x7fc2dc13f710 2024-03-25T19:39:00.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.912+0000 7fc2d3fff640 1 -- 172.21.15.19:0/2549231053 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197951225 0 0) 0x7fc2dc0b28a0 con 0x7fc2dc1430c0 2024-03-25T19:39:00.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.912+0000 7fc2d3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.912+0000 7fc2d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.912+0000 7fc2d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.912+0000 7fc2d3fff640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:39:00.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.912+0000 7fc2d3fff640 1 -- 172.21.15.19:0/2549231053 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2dc0b2a30 con 0x7fc2dc1430c0 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.918+0000 7fc2d3fff640 1 -- 172.21.15.19:0/2549231053 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1251597533 0 0) 0x7fc2c8003520 con 0x7fc2dc13f710 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.918+0000 7fc2d3fff640 1 -- 172.21.15.19:0/2549231053 >> v1:172.21.15.188:6789/0 conn(0x7fc2dc1430c0 legacy=0x7fc2dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.918+0000 7fc2d3fff640 1 -- 172.21.15.19:0/2549231053 >> v1:172.21.15.19:6789/0 conn(0x7fc2dc0a98f0 legacy=0x7fc2dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.918+0000 7fc2d3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.918+0000 7fc2d3fff640 1 monclient: found mon.noname-b 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.918+0000 7fc2d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.918+0000 7fc2d3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.918+0000 7fc2d3fff640 1 -- 172.21.15.19:0/2549231053 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc0b44c0 con 0x7fc2dc13f710 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.918+0000 7fc2d3fff640 10 monclient: _finish_auth 0 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.919+0000 7fc2d3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.919+0000 7fc2d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.919+0000 7fc2d3fff640 1 -- 172.21.15.19:0/2549231053 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc2cc001690 con 0x7fc2dc13f710 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.919+0000 7fc2d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:00.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.919+0000 7fc2d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.919+0000 7fc2d3fff640 10 monclient: dump: 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.919+0000 7fc2e34ff640 5 monclient: authenticate success, global_id 4643 2024-03-25T19:39:00.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.919+0000 7fc2e34ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:00.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.931+0000 7fc2d3fff640 1 -- 172.21.15.19:0/2549231053 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2cc0017f0 con 0x7fc2dc13f710 2024-03-25T19:39:00.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.931+0000 7fc2d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:00.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.931+0000 7fc2d3fff640 1 -- 172.21.15.19:0/2549231053 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc2cc001c00 con 0x7fc2dc13f710 2024-03-25T19:39:00.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.931+0000 7fc2e34ff640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:00.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.931+0000 7fc2e34ff640 4 set_mon_vals no callback set 2024-03-25T19:39:00.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.931+0000 7fc2d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:00.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.931+0000 7fc2d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:00.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.931+0000 7fc2d3fff640 10 monclient: dump: 2024-03-25T19:39:00.931 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:00.931 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:00.931 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:00.931 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:00.932 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:00.932 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:00.932 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:00.932 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:00.932 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:00.932 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:00.933 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.932+0000 7fc2e34ff640 10 monclient: shutdown 2024-03-25T19:39:00.933 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.932+0000 7fc2e34ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:00.933 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.932+0000 7fc2e34ff640 1 -- 172.21.15.19:0/2549231053 >> v1:172.21.15.19:6790/0 conn(0x7fc2dc13f710 legacy=0x7fc2dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:00.933 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.932+0000 7fc2e34ff640 1 -- 172.21.15.19:0/2549231053 shutdown_connections 2024-03-25T19:39:00.933 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.932+0000 7fc2e34ff640 1 -- 172.21.15.19:0/2549231053 >> 172.21.15.19:0/2549231053 conn(0x7fc2dc0b7ea0 msgr2=0x7fc2dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:00.933 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.932+0000 7fc2e34ff640 1 -- 172.21.15.19:0/2549231053 shutdown_connections 2024-03-25T19:39:00.933 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.932+0000 7fc2e34ff640 1 -- 172.21.15.19:0/2549231053 wait complete. 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.933+0000 7fc2e34ff640 10 monclient: build_initial_monmap 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.933+0000 7fc2e34ff640 10 monclient: monmap: 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.933+0000 7fc2e34ff640 1 librados: starting msgr at 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.933+0000 7fc2e34ff640 1 librados: starting objecter 2024-03-25T19:39:00.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.933+0000 7fc2e34ff640 1 Processor -- start 2024-03-25T19:39:00.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.933+0000 7fc2e34ff640 1 -- start start 2024-03-25T19:39:00.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.933+0000 7fc2e34ff640 1 librados: setting wanted keys 2024-03-25T19:39:00.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.933+0000 7fc2e34ff640 1 librados: calling monclient init 2024-03-25T19:39:00.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.933+0000 7fc2e34ff640 10 monclient: init 2024-03-25T19:39:00.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.934+0000 7fc2e34ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:00.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.934+0000 7fc2e34ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:00.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.934+0000 7fc2e34ff640 10 monclient(hunting): picked mon.noname-b con 0x7fc2dc0a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:00.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.934+0000 7fc2e34ff640 10 monclient(hunting): picked mon.noname-c con 0x7fc2dc13f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:00.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.934+0000 7fc2e34ff640 10 monclient(hunting): picked mon.noname-a con 0x7fc2dc1430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:00.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.934+0000 7fc2e34ff640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc137df0 con 0x7fc2dc1430c0 2024-03-25T19:39:00.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.934+0000 7fc2e34ff640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc1337a0 con 0x7fc2dc0a98f0 2024-03-25T19:39:00.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.934+0000 7fc2e34ff640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc133930 con 0x7fc2dc13f710 2024-03-25T19:39:00.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.934+0000 7fc2e34ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:00.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.934+0000 7fc2e34ff640 10 monclient(hunting): authenticate will time out at 6594.204359s 2024-03-25T19:39:00.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.937+0000 7fc2e0a9a640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fc2dc13f710 0x7fc2dc1376e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:52132/0 (socket says 172.21.15.19:52132) 2024-03-25T19:39:00.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.937+0000 7fc2e0a9a640 1 -- 172.21.15.19:0/1375330534 learned_addr learned my addr 172.21.15.19:0/1375330534 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:00.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.939+0000 7fc2d1ffb640 10 client.?.objecter ms_handle_connect 0x7fc2dc13f710 2024-03-25T19:39:00.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.939+0000 7fc2d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:00.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.946+0000 7fc2d1ffb640 10 client.?.objecter ms_handle_connect 0x7fc2dc0a98f0 2024-03-25T19:39:00.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.946+0000 7fc2d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.954+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466430667 0 0) 0x7fc2dc133930 con 0x7fc2dc13f710 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.954+0000 7fc2d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.954+0000 7fc2d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.954+0000 7fc2d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.954+0000 7fc2d1ffb640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:39:00.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.954+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c0004eb0 con 0x7fc2dc13f710 2024-03-25T19:39:00.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.956+0000 7fc2d1ffb640 10 client.?.objecter ms_handle_connect 0x7fc2dc1430c0 2024-03-25T19:39:00.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.956+0000 7fc2d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:00.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.956+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332025902 0 0) 0x7fc2dc1337a0 con 0x7fc2dc0a98f0 2024-03-25T19:39:00.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.956+0000 7fc2d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.956+0000 7fc2d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.956+0000 7fc2d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.956+0000 7fc2d1ffb640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:39:00.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.956+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2dc133930 con 0x7fc2dc0a98f0 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.972+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3780726570 0 0) 0x7fc2dc133930 con 0x7fc2dc0a98f0 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.972+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2dc1337a0 con 0x7fc2dc0a98f0 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.972+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc2d80018b0 con 0x7fc2dc0a98f0 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.972+0000 7fc2d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.972+0000 7fc2d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.972+0000 7fc2d1ffb640 10 monclient(hunting): dump: 2024-03-25T19:39:00.972 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:00.973 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:00.973 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:00.973 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:00.973 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:00.973 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:00.973 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:00.973 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:00.973 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:00.973 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:00.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.972+0000 7fc2d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:00.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.975+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2632711760 0 0) 0x7fc2c0004eb0 con 0x7fc2dc13f710 2024-03-25T19:39:00.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.975+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2dc133930 con 0x7fc2dc13f710 2024-03-25T19:39:00.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.975+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc2cc001bc0 con 0x7fc2dc13f710 2024-03-25T19:39:00.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.975+0000 7fc2d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:00.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.975+0000 7fc2d1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:00.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.975+0000 7fc2d1ffb640 10 monclient(hunting): dump: 2024-03-25T19:39:00.975 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:00.975 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:00.975 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:00.976 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:00.976 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:00.976 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:00.976 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:00.976 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:00.976 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:00.976 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:00.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.976+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693854825 0 0) 0x7fc2dc137df0 con 0x7fc2dc1430c0 2024-03-25T19:39:00.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.976+0000 7fc2d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:00.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.976+0000 7fc2d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:00.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.976+0000 7fc2d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:00.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.976+0000 7fc2d1ffb640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:39:00.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.976+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c0004eb0 con 0x7fc2dc1430c0 2024-03-25T19:39:00.980 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.979+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2284154703 0 0) 0x7fc2dc1337a0 con 0x7fc2dc0a98f0 2024-03-25T19:39:00.980 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.979+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 >> v1:172.21.15.19:6790/0 conn(0x7fc2dc13f710 legacy=0x7fc2dc1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:00.980 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.979+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 >> v1:172.21.15.19:6789/0 conn(0x7fc2dc1430c0 legacy=0x7fc2dc133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:00.980 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.979+0000 7fc2d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:39:00.980 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.979+0000 7fc2d1ffb640 1 monclient: found mon.b 2024-03-25T19:39:00.980 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.979+0000 7fc2d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:00.980 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.979+0000 7fc2d1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:39:00.980 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.979+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc1367d0 con 0x7fc2dc0a98f0 2024-03-25T19:39:00.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.980+0000 7fc2d1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:39:00.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.980+0000 7fc2d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:39:00.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.980+0000 7fc2d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:00.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.980+0000 7fc2e34ff640 5 monclient: authenticate success, global_id 4669 2024-03-25T19:39:00.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.980+0000 7fc2e34ff640 10 monclient: _renew_subs 2024-03-25T19:39:00.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.980+0000 7fc2e34ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:39:00.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.980+0000 7fc2e34ff640 1 -- 172.21.15.19:0/1375330534 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2dc1369e0 con 0x7fc2dc0a98f0 2024-03-25T19:39:00.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.980+0000 7fc2e34ff640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:00.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.980+0000 7fc2e34ff640 10 monclient: _renew_subs 2024-03-25T19:39:00.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.980+0000 7fc2e34ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:39:00.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.980+0000 7fc2e34ff640 1 -- 172.21.15.19:0/1375330534 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc13b640 con 0x7fc2dc0a98f0 2024-03-25T19:39:00.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.980+0000 7fc2e34ff640 1 librados: init done 2024-03-25T19:39:00.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.981+0000 7fc2e34ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:00.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.981+0000 7fc2e34ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:00.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.981+0000 7fc2e34ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:39:00.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:00.981+0000 7fc2e34ff640 1 -- 172.21.15.19:0/1375330534 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2dc0627b0 con 0x7fc2dc0a98f0 2024-03-25T19:39:01.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.264+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2d8003610 con 0x7fc2dc0a98f0 2024-03-25T19:39:01.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.264+0000 7fc2d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:01.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.264+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc2d8003a60 con 0x7fc2dc0a98f0 2024-03-25T19:39:01.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.264+0000 7fc2d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:01.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.264+0000 7fc2d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:01.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.264+0000 7fc2d1ffb640 10 monclient: dump: 2024-03-25T19:39:01.264 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:01.264 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:01.264 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:01.265 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:01.265 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:01.265 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:01.265 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:01.265 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:01.265 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:01.265 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:01.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.264+0000 7fc2d37fe640 4 set_mon_vals no callback set 2024-03-25T19:39:01.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.265+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fc2d8003c80 con 0x7fc2dc0a98f0 2024-03-25T19:39:01.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.266+0000 7fc2e0a9a640 10 monclient: get_auth_request con 0x7fc2c006df40 auth_method 0 2024-03-25T19:39:01.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.266+0000 7fc2d1ffb640 10 client.4669.objecter ms_handle_connect 0x7fc2c006df40 2024-03-25T19:39:01.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.270+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1490778728 0 0) 0x7fc2d80535b0 con 0x7fc2dc0a98f0 2024-03-25T19:39:01.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.270+0000 7fc2d1ffb640 10 client.4669.objecter ms_dispatch 0x7fc2dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:01.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.270+0000 7fc2d1ffb640 3 client.4669.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:01.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.270+0000 7fc2d1ffb640 3 client.4669.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:01.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.270+0000 7fc2d1ffb640 20 client.4669.objecter dump_active .. 0 homeless 2024-03-25T19:39:01.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.270+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2d804f0c0 con 0x7fc2dc0a98f0 2024-03-25T19:39:01.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.270+0000 7fc2d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:01.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.270+0000 7fc2d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:01.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.485+0000 7fc2e34ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:01.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.485+0000 7fc2e34ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:01.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.485+0000 7fc2e34ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:39:01.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.485+0000 7fc2e34ff640 1 -- 172.21.15.19:0/1375330534 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2dc074cc0 con 0x7fc2dc0a98f0 2024-03-25T19:39:01.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.516+0000 7fc2d1ffb640 1 -- 172.21.15.19:0/1375330534 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1576595288) 0x7fc2d8050ff0 con 0x7fc2dc0a98f0 2024-03-25T19:39:01.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.516+0000 7fc2d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:01.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.516+0000 7fc2d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:01.516 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:39:01.516 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:38:58.470042+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:58.470042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 10 librados: watch_flush enter 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 10 librados: watch_flush exit 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 20 client.4669.objecter shutdown clearing up homeless session... 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 10 client.4669.objecter successfully canceled tick 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 1 -- 172.21.15.19:0/1375330534 >> v1:172.21.15.188:6800/2207506229 conn(0x7fc2c006df40 legacy=0x7fc2c0070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 10 monclient: shutdown 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 1 -- 172.21.15.19:0/1375330534 >> v1:172.21.15.188:6789/0 conn(0x7fc2dc0a98f0 legacy=0x7fc2dc1392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 1 -- 172.21.15.19:0/1375330534 shutdown_connections 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 1 -- 172.21.15.19:0/1375330534 >> 172.21.15.19:0/1375330534 conn(0x7fc2dc0b7ea0 msgr2=0x7fc2dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 1 -- 172.21.15.19:0/1375330534 shutdown_connections 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 1 -- 172.21.15.19:0/1375330534 wait complete. 2024-03-25T19:39:01.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:01.518+0000 7fc2e34ff640 1 librados: shutdown 2024-03-25T19:39:01.530 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:08.336 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:08.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.453+0000 7fbf197c8640 10 monclient: get_monmap_and_config 2024-03-25T19:39:08.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.453+0000 7fbf197c8640 10 monclient: build_initial_monmap 2024-03-25T19:39:08.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.453+0000 7fbf197c8640 10 monclient: monmap: 2024-03-25T19:39:08.453 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:08.453 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:08.453 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:39:08.454813+0000 2024-03-25T19:39:08.454 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:39:08.454813+0000 2024-03-25T19:39:08.454 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:08.454 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:08.454 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:08.454 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:39:08.454 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:39:08.454 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:08.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.454+0000 7fbf197c8640 1 Processor -- start 2024-03-25T19:39:08.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.454+0000 7fbf197c8640 1 -- start start 2024-03-25T19:39:08.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.454+0000 7fbf197c8640 10 monclient: init 2024-03-25T19:39:08.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.455+0000 7fbf197c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:08.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.455+0000 7fbf197c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:08.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.455+0000 7fbf197c8640 10 monclient(hunting): picked mon.noname-c con 0x7fbf14061830 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:08.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.455+0000 7fbf197c8640 10 monclient(hunting): picked mon.noname-a con 0x7fbf1413fcd0 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:08.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.455+0000 7fbf197c8640 10 monclient(hunting): picked mon.noname-b con 0x7fbf140ae840 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:08.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.455+0000 7fbf197c8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf1406f760 con 0x7fbf1413fcd0 2024-03-25T19:39:08.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.455+0000 7fbf197c8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf1406fa40 con 0x7fbf14061830 2024-03-25T19:39:08.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.455+0000 7fbf197c8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf140ae0c0 con 0x7fbf140ae840 2024-03-25T19:39:08.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.455+0000 7fbf197c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:08.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.455+0000 7fbf197c8640 10 monclient(hunting): authenticate will time out at 6601.725243s 2024-03-25T19:39:08.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.459+0000 7fbf12ffd640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fbf14061830 0x7fbf1413f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:57042/0 (socket says 172.21.15.19:57042) 2024-03-25T19:39:08.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.459+0000 7fbf12ffd640 1 -- 172.21.15.19:0/1378387358 learned_addr learned my addr 172.21.15.19:0/1378387358 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:08.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.468+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464692677 0 0) 0x7fbf1406fa40 con 0x7fbf14061830 2024-03-25T19:39:08.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.468+0000 7fbf11ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.468+0000 7fbf11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.468+0000 7fbf11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.468+0000 7fbf11ffb640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:39:08.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.468+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbefc003520 con 0x7fbf14061830 2024-03-25T19:39:08.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.492+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734494281 0 0) 0x7fbf1406f760 con 0x7fbf1413fcd0 2024-03-25T19:39:08.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.492+0000 7fbf11ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.492+0000 7fbf11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.492+0000 7fbf11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.492+0000 7fbf11ffb640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:39:08.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.492+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf1406fa40 con 0x7fbf1413fcd0 2024-03-25T19:39:08.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091292879 0 0) 0x7fbf140ae0c0 con 0x7fbf140ae840 2024-03-25T19:39:08.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:39:08.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf1406f760 con 0x7fbf140ae840 2024-03-25T19:39:08.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2125144760 0 0) 0x7fbefc003520 con 0x7fbf14061830 2024-03-25T19:39:08.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 >> v1:172.21.15.19:6790/0 conn(0x7fbf140ae840 legacy=0x7fbf14145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 >> v1:172.21.15.19:6789/0 conn(0x7fbf1413fcd0 legacy=0x7fbf14145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 1 monclient: found mon.noname-c 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf14073040 con 0x7fbf14061830 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 10 monclient: _finish_auth 0 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fbf04001690 con 0x7fbf14061830 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf11ffb640 10 monclient: dump: 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:08.496 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:08.497 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:08.497 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:08.497 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:08.497 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:08.497 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:08.497 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:08.497 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:08.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf197c8640 5 monclient: authenticate success, global_id 4672 2024-03-25T19:39:08.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.494+0000 7fbf197c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:08.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.509+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf040017f0 con 0x7fbf14061830 2024-03-25T19:39:08.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.509+0000 7fbf11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:08.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.509+0000 7fbf11ffb640 1 -- 172.21.15.19:0/1378387358 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fbf04001c00 con 0x7fbf14061830 2024-03-25T19:39:08.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.509+0000 7fbf197c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:08.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.509+0000 7fbf197c8640 4 set_mon_vals no callback set 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.509+0000 7fbf11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.509+0000 7fbf11ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.509+0000 7fbf11ffb640 10 monclient: dump: 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:08.510 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:08.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.510+0000 7fbf197c8640 10 monclient: shutdown 2024-03-25T19:39:08.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.510+0000 7fbf197c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:08.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.510+0000 7fbf197c8640 1 -- 172.21.15.19:0/1378387358 >> v1:172.21.15.188:6789/0 conn(0x7fbf14061830 legacy=0x7fbf1413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.510+0000 7fbf197c8640 1 -- 172.21.15.19:0/1378387358 shutdown_connections 2024-03-25T19:39:08.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.510+0000 7fbf197c8640 1 -- 172.21.15.19:0/1378387358 >> 172.21.15.19:0/1378387358 conn(0x7fbf140b0ef0 msgr2=0x7fbf140b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:08.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.510+0000 7fbf197c8640 1 -- 172.21.15.19:0/1378387358 shutdown_connections 2024-03-25T19:39:08.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.510+0000 7fbf197c8640 1 -- 172.21.15.19:0/1378387358 wait complete. 2024-03-25T19:39:08.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.511+0000 7fbf197c8640 10 monclient: build_initial_monmap 2024-03-25T19:39:08.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.511+0000 7fbf197c8640 10 monclient: monmap: 2024-03-25T19:39:08.512 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:08.512 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.511+0000 7fbf197c8640 1 librados: starting msgr at 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.511+0000 7fbf197c8640 1 librados: starting objecter 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 1 Processor -- start 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 1 -- start start 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 1 librados: setting wanted keys 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 1 librados: calling monclient init 2024-03-25T19:39:08.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 10 monclient: init 2024-03-25T19:39:08.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:08.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:08.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 10 monclient(hunting): picked mon.noname-b con 0x7fbf14061830 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:08.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 10 monclient(hunting): picked mon.noname-c con 0x7fbf140ae840 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:08.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 10 monclient(hunting): picked mon.noname-a con 0x7fbf1413fcd0 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:08.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf1413b000 con 0x7fbf1413fcd0 2024-03-25T19:39:08.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf1413b2e0 con 0x7fbf14061830 2024-03-25T19:39:08.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf14138e60 con 0x7fbf140ae840 2024-03-25T19:39:08.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:08.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.512+0000 7fbf197c8640 10 monclient(hunting): authenticate will time out at 6601.782877s 2024-03-25T19:39:08.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.525+0000 7fbf12ffd640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fbf14061830 0x7fbf14139620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:57088/0 (socket says 172.21.15.19:57088) 2024-03-25T19:39:08.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.525+0000 7fbf12ffd640 1 -- 172.21.15.19:0/3043136332 learned_addr learned my addr 172.21.15.19:0/3043136332 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:08.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.528+0000 7fbefb7fe640 10 client.?.objecter ms_handle_connect 0x7fbf14061830 2024-03-25T19:39:08.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.528+0000 7fbefb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:08.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.535+0000 7fbefb7fe640 10 client.?.objecter ms_handle_connect 0x7fbf1413fcd0 2024-03-25T19:39:08.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.535+0000 7fbefb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:08.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.535+0000 7fbefb7fe640 10 client.?.objecter ms_handle_connect 0x7fbf140ae840 2024-03-25T19:39:08.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.535+0000 7fbefb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:08.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.539+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264958042 0 0) 0x7fbf1413b2e0 con 0x7fbf14061830 2024-03-25T19:39:08.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.539+0000 7fbefb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.539+0000 7fbefb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.539+0000 7fbefb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.539+0000 7fbefb7fe640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:39:08.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.539+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf00004eb0 con 0x7fbf14061830 2024-03-25T19:39:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.542+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380591009 0 0) 0x7fbf14138e60 con 0x7fbf140ae840 2024-03-25T19:39:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.542+0000 7fbefb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.542+0000 7fbefb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.542+0000 7fbefb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:08.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.542+0000 7fbefb7fe640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:39:08.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.542+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf1413b2e0 con 0x7fbf140ae840 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.550+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2167041746 0 0) 0x7fbf00004eb0 con 0x7fbf14061830 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.550+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf14138e60 con 0x7fbf14061830 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.550+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fbf04001bc0 con 0x7fbf14061830 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.550+0000 7fbefb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.550+0000 7fbefb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.550+0000 7fbefb7fe640 10 monclient(hunting): dump: 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:08.550 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:08.551 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:08.551 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:08.551 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:08.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.550+0000 7fbefb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:08.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2644107992 0 0) 0x7fbf14138e60 con 0x7fbf14061830 2024-03-25T19:39:08.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 >> v1:172.21.15.19:6790/0 conn(0x7fbf140ae840 legacy=0x7fbf14136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 >> v1:172.21.15.19:6789/0 conn(0x7fbf1413fcd0 legacy=0x7fbf141e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbefb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbefb7fe640 1 monclient: found mon.b 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbefb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbefb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf14139e80 con 0x7fbf14061830 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbefb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbefb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbefb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbf197c8640 5 monclient: authenticate success, global_id 4675 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbf197c8640 10 monclient: _renew_subs 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbf197c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbf197c8640 1 -- 172.21.15.19:0/3043136332 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf14136790 con 0x7fbf14061830 2024-03-25T19:39:08.562 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbf197c8640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:08.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbf197c8640 10 monclient: _renew_subs 2024-03-25T19:39:08.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbf197c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:39:08.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbf197c8640 1 -- 172.21.15.19:0/3043136332 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf141e3d80 con 0x7fbf14061830 2024-03-25T19:39:08.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.561+0000 7fbf197c8640 1 librados: init done 2024-03-25T19:39:08.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.562+0000 7fbf197c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:08.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.562+0000 7fbf197c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:08.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.562+0000 7fbf197c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:39:08.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.562+0000 7fbf197c8640 1 -- 172.21.15.19:0/3043136332 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf14068290 con 0x7fbf14061830 2024-03-25T19:39:08.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.867+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf040011f0 con 0x7fbf14061830 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.867+0000 7fbefb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.867+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fbf04004c30 con 0x7fbf14061830 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.867+0000 7fbefb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.867+0000 7fbefb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.867+0000 7fbefb7fe640 10 monclient: dump: 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:08.868 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:08.869 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:08.869 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:08.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.867+0000 7fbf117fa640 4 set_mon_vals no callback set 2024-03-25T19:39:08.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.868+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fbf0401ac40 con 0x7fbf14061830 2024-03-25T19:39:08.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.869+0000 7fbf127fc640 10 monclient: get_auth_request con 0x7fbf0006dc70 auth_method 0 2024-03-25T19:39:08.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.869+0000 7fbefb7fe640 10 client.4675.objecter ms_handle_connect 0x7fbf0006dc70 2024-03-25T19:39:08.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.874+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1490778728 0 0) 0x7fbf04054930 con 0x7fbf14061830 2024-03-25T19:39:08.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.874+0000 7fbefb7fe640 10 client.4675.objecter ms_dispatch 0x7fbf14000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:08.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.874+0000 7fbefb7fe640 3 client.4675.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:08.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.874+0000 7fbefb7fe640 3 client.4675.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:08.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.874+0000 7fbefb7fe640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T19:39:08.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.874+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf04050270 con 0x7fbf14061830 2024-03-25T19:39:08.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.874+0000 7fbefb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:08.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:08.874+0000 7fbefb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:09.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.112+0000 7fbf197c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:09.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.112+0000 7fbf197c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:09.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.112+0000 7fbf197c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:39:09.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.112+0000 7fbf197c8640 1 -- 172.21.15.19:0/3043136332 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf140627b0 con 0x7fbf14061830 2024-03-25T19:39:09.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.191+0000 7fbefb7fe640 1 -- 172.21.15.19:0/3043136332 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1576595288) 0x7fbf04052370 con 0x7fbf14061830 2024-03-25T19:39:09.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.191+0000 7fbefb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:09.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.191+0000 7fbefb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:09.191 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:39:09.192 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:38:58.470042+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:58.470042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:09.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 10 librados: watch_flush enter 2024-03-25T19:39:09.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 10 librados: watch_flush exit 2024-03-25T19:39:09.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T19:39:09.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 10 client.4675.objecter successfully canceled tick 2024-03-25T19:39:09.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 1 -- 172.21.15.19:0/3043136332 >> v1:172.21.15.188:6800/2207506229 conn(0x7fbf0006dc70 legacy=0x7fbf00070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:09.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 10 monclient: shutdown 2024-03-25T19:39:09.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:09.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 1 -- 172.21.15.19:0/3043136332 >> v1:172.21.15.188:6789/0 conn(0x7fbf14061830 legacy=0x7fbf14139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:09.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 1 -- 172.21.15.19:0/3043136332 shutdown_connections 2024-03-25T19:39:09.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 1 -- 172.21.15.19:0/3043136332 >> 172.21.15.19:0/3043136332 conn(0x7fbf140b0ef0 msgr2=0x7fbf140b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:09.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 1 -- 172.21.15.19:0/3043136332 shutdown_connections 2024-03-25T19:39:09.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 1 -- 172.21.15.19:0/3043136332 wait complete. 2024-03-25T19:39:09.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:09.194+0000 7fbf197c8640 1 librados: shutdown 2024-03-25T19:39:09.205 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:16.013 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:16.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.125+0000 7f20abf5d640 10 monclient: get_monmap_and_config 2024-03-25T19:39:16.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.125+0000 7f20abf5d640 10 monclient: build_initial_monmap 2024-03-25T19:39:16.125 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.125+0000 7f20abf5d640 10 monclient: monmap: 2024-03-25T19:39:16.125 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:16.125 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:16.125 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:39:16.126641+0000 2024-03-25T19:39:16.126 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:39:16.126641+0000 2024-03-25T19:39:16.126 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:16.126 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:16.126 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:16.126 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:39:16.126 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:39:16.126 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:16.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 1 Processor -- start 2024-03-25T19:39:16.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 1 -- start start 2024-03-25T19:39:16.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 10 monclient: init 2024-03-25T19:39:16.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:16.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:16.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 10 monclient(hunting): picked mon.noname-b con 0x7f20a40b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:16.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 10 monclient(hunting): picked mon.noname-c con 0x7f20a40b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:16.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 10 monclient(hunting): picked mon.noname-a con 0x7f20a414be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:16.127 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a40af940 con 0x7f20a414be90 2024-03-25T19:39:16.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a40b0a70 con 0x7f20a40b1730 2024-03-25T19:39:16.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a40b0d50 con 0x7f20a40b4d60 2024-03-25T19:39:16.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:16.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.126+0000 7f20abf5d640 10 monclient(hunting): authenticate will time out at 6609.396829s 2024-03-25T19:39:16.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.137+0000 7f20a94f8640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f20a40b1730 0x7f20a40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:33566/0 (socket says 172.21.15.19:33566) 2024-03-25T19:39:16.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.137+0000 7f20a94f8640 1 -- 172.21.15.19:0/2965269859 learned_addr learned my addr 172.21.15.19:0/2965269859 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:16.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.144+0000 7f20a8cf7640 1 -- 172.21.15.19:0/2965269859 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102430267 0 0) 0x7f20a40b0a70 con 0x7f20a40b1730 2024-03-25T19:39:16.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.144+0000 7f20a8cf7640 10 monclient(hunting): _negotiate 2024-03-25T19:39:16.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.144+0000 7f20a8cf7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:16.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.144+0000 7f20a8cf7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:16.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.144+0000 7f20a8cf7640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:39:16.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.144+0000 7f20a8cf7640 1 -- 172.21.15.19:0/2965269859 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2090003520 con 0x7f20a40b1730 2024-03-25T19:39:16.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 1 -- 172.21.15.19:0/2965269859 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1386220023 0 0) 0x7f2090003520 con 0x7f20a40b1730 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 1 -- 172.21.15.19:0/2965269859 >> v1:172.21.15.19:6790/0 conn(0x7f20a40b4d60 legacy=0x7f20a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 1 -- 172.21.15.19:0/2965269859 >> v1:172.21.15.19:6789/0 conn(0x7f20a414be90 legacy=0x7f20a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 10 monclient: _finish_hunting 0 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 1 monclient: found mon.noname-c 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 1 -- 172.21.15.19:0/2965269859 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a40b1030 con 0x7f20a40b1730 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 10 monclient: _finish_auth 0 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 10 monclient: _check_auth_tickets 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 1 -- 172.21.15.19:0/2965269859 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2094001690 con 0x7f20a40b1730 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20a8cf7640 10 monclient: dump: 2024-03-25T19:39:16.152 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:16.153 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:16.153 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:16.153 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:16.153 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:16.153 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:16.153 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:16.153 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:16.153 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:16.153 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:16.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20abf5d640 5 monclient: authenticate success, global_id 4678 2024-03-25T19:39:16.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.151+0000 7f20abf5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:16.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.161+0000 7f20a8cf7640 1 -- 172.21.15.19:0/2965269859 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2094001860 con 0x7f20a40b1730 2024-03-25T19:39:16.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.161+0000 7f20a8cf7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:16.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.161+0000 7f20a8cf7640 1 -- 172.21.15.19:0/2965269859 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2094001cb0 con 0x7f20a40b1730 2024-03-25T19:39:16.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.161+0000 7f20abf5d640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:16.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.161+0000 7f20abf5d640 4 set_mon_vals no callback set 2024-03-25T19:39:16.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.161+0000 7f20a8cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:16.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.161+0000 7f20a8cf7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:16.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.161+0000 7f20a8cf7640 10 monclient: dump: 2024-03-25T19:39:16.161 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:16.161 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:16.162 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:16.162 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:16.162 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:16.162 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:16.162 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:16.162 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:16.162 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:16.162 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:16.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.162+0000 7f20abf5d640 10 monclient: shutdown 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.162+0000 7f20abf5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.162+0000 7f20abf5d640 1 -- 172.21.15.19:0/2965269859 >> v1:172.21.15.188:6789/0 conn(0x7f20a40b1730 legacy=0x7f20a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.162+0000 7f20abf5d640 1 -- 172.21.15.19:0/2965269859 shutdown_connections 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.162+0000 7f20abf5d640 1 -- 172.21.15.19:0/2965269859 >> 172.21.15.19:0/2965269859 conn(0x7f20a40617a0 msgr2=0x7f20a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.162+0000 7f20abf5d640 1 -- 172.21.15.19:0/2965269859 shutdown_connections 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.162+0000 7f20abf5d640 1 -- 172.21.15.19:0/2965269859 wait complete. 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.163+0000 7f20abf5d640 10 monclient: build_initial_monmap 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.163+0000 7f20abf5d640 10 monclient: monmap: 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:39:16.164 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:16.165 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:39:16.165 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:39:16.165 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:16.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.163+0000 7f20abf5d640 1 librados: starting msgr at 2024-03-25T19:39:16.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.163+0000 7f20abf5d640 1 librados: starting objecter 2024-03-25T19:39:16.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.163+0000 7f20abf5d640 1 Processor -- start 2024-03-25T19:39:16.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.163+0000 7f20abf5d640 1 -- start start 2024-03-25T19:39:16.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.163+0000 7f20abf5d640 1 librados: setting wanted keys 2024-03-25T19:39:16.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.163+0000 7f20abf5d640 1 librados: calling monclient init 2024-03-25T19:39:16.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.163+0000 7f20abf5d640 10 monclient: init 2024-03-25T19:39:16.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.164+0000 7f20abf5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:16.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.164+0000 7f20abf5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:16.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.164+0000 7f20abf5d640 10 monclient(hunting): picked mon.noname-b con 0x7f20a40b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:16.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.164+0000 7f20abf5d640 10 monclient(hunting): picked mon.noname-c con 0x7f20a40b4d60 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:16.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.164+0000 7f20abf5d640 10 monclient(hunting): picked mon.noname-a con 0x7f20a414be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:16.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.164+0000 7f20abf5d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a4142220 con 0x7f20a414be90 2024-03-25T19:39:16.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.164+0000 7f20abf5d640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a41377a0 con 0x7f20a40b1730 2024-03-25T19:39:16.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.164+0000 7f20abf5d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a4137930 con 0x7f20a40b4d60 2024-03-25T19:39:16.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.164+0000 7f20abf5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:16.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.164+0000 7f20abf5d640 10 monclient(hunting): authenticate will time out at 6609.434840s 2024-03-25T19:39:16.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.177+0000 7f20a94f8640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f20a40b4d60 0x7f20a41dab60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:50946/0 (socket says 172.21.15.19:50946) 2024-03-25T19:39:16.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.177+0000 7f20a94f8640 1 -- 172.21.15.19:0/1441571020 learned_addr learned my addr 172.21.15.19:0/1441571020 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:16.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.179+0000 7f209a7fc640 10 client.?.objecter ms_handle_connect 0x7f20a40b4d60 2024-03-25T19:39:16.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.179+0000 7f209a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:16.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.179+0000 7f209a7fc640 10 client.?.objecter ms_handle_connect 0x7f20a40b1730 2024-03-25T19:39:16.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.179+0000 7f209a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:16.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.182+0000 7f209a7fc640 10 client.?.objecter ms_handle_connect 0x7f20a414be90 2024-03-25T19:39:16.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.182+0000 7f209a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:16.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.188+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611908021 0 0) 0x7f20a4137930 con 0x7f20a40b4d60 2024-03-25T19:39:16.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.188+0000 7f209a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:16.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.188+0000 7f209a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:16.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.188+0000 7f209a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:16.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.188+0000 7f209a7fc640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:39:16.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.188+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2088004eb0 con 0x7f20a40b4d60 2024-03-25T19:39:16.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.188+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268940668 0 0) 0x7f20a4142220 con 0x7f20a414be90 2024-03-25T19:39:16.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.188+0000 7f209a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:16.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.188+0000 7f209a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:16.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.188+0000 7f209a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:16.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.188+0000 7f209a7fc640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:39:16.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.188+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a4137930 con 0x7f20a414be90 2024-03-25T19:39:16.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.199+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743044012 0 0) 0x7f20a41377a0 con 0x7f20a40b1730 2024-03-25T19:39:16.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.199+0000 7f209a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:16.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.199+0000 7f209a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:16.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.199+0000 7f209a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:16.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.199+0000 7f209a7fc640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:39:16.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.199+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a4142220 con 0x7f20a40b1730 2024-03-25T19:39:16.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.200+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 618577966 0 0) 0x7f20a4137930 con 0x7f20a414be90 2024-03-25T19:39:16.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.200+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20a41377a0 con 0x7f20a414be90 2024-03-25T19:39:16.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.200+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f209c001a70 con 0x7f20a414be90 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.200+0000 7f209a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.200+0000 7f209a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.200+0000 7f209a7fc640 10 monclient(hunting): dump: 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:16.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.200+0000 7f209a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:16.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.206+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1196546310 0 0) 0x7f2088004eb0 con 0x7f20a40b4d60 2024-03-25T19:39:16.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.206+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20a4137930 con 0x7f20a40b4d60 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.206+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f2094001b70 con 0x7f20a40b4d60 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.206+0000 7f209a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.206+0000 7f209a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.206+0000 7f209a7fc640 10 monclient(hunting): dump: 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:16.207 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:16.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.211+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1119907194 0 0) 0x7f20a4142220 con 0x7f20a40b1730 2024-03-25T19:39:16.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.211+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2088004eb0 con 0x7f20a40b1730 2024-03-25T19:39:16.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.211+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f20a0001830 con 0x7f20a40b1730 2024-03-25T19:39:16.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.211+0000 7f209a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:16.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.211+0000 7f209a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:16.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.211+0000 7f209a7fc640 10 monclient(hunting): dump: 2024-03-25T19:39:16.212 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:16.212 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:16.212 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:16.212 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:16.212 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:16.212 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:16.212 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:16.212 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:16.212 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:16.212 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:16.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 716289295 0 0) 0x7f20a41377a0 con 0x7f20a414be90 2024-03-25T19:39:16.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 >> v1:172.21.15.19:6790/0 conn(0x7f20a40b4d60 legacy=0x7f20a41dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 >> v1:172.21.15.188:6789/0 conn(0x7f20a40b1730 legacy=0x7f20a4137090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f209a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:39:16.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f209a7fc640 1 monclient: found mon.a 2024-03-25T19:39:16.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f209a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:16.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f209a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:16.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a41416d0 con 0x7f20a414be90 2024-03-25T19:39:16.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f209a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:39:16.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f209a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:39:16.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f209a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:16.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f20abf5d640 5 monclient: authenticate success, global_id 4683 2024-03-25T19:39:16.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f20abf5d640 10 monclient: _renew_subs 2024-03-25T19:39:16.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f20abf5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:16.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f20abf5d640 1 -- 172.21.15.19:0/1441571020 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a4141920 con 0x7f20a414be90 2024-03-25T19:39:16.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f20abf5d640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:16.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f20abf5d640 10 monclient: _renew_subs 2024-03-25T19:39:16.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f20abf5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:16.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f20abf5d640 1 -- 172.21.15.19:0/1441571020 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a413ab00 con 0x7f20a414be90 2024-03-25T19:39:16.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.215+0000 7f20abf5d640 1 librados: init done 2024-03-25T19:39:16.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.216+0000 7f20abf5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:16.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.216+0000 7f20abf5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:16.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.216+0000 7f20abf5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:16.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.216+0000 7f20abf5d640 1 -- 172.21.15.19:0/1441571020 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a40a9ab0 con 0x7f20a414be90 2024-03-25T19:39:16.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.548+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f209c001f60 con 0x7f20a414be90 2024-03-25T19:39:16.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.548+0000 7f209a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:16.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.548+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f209c003b10 con 0x7f20a414be90 2024-03-25T19:39:16.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.548+0000 7f209a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:16.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.548+0000 7f209a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:16.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.548+0000 7f209a7fc640 10 monclient: dump: 2024-03-25T19:39:16.549 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:16.549 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:16.549 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:16.549 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:16.549 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:16.550 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:16.550 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:16.550 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:16.550 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:16.550 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:16.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.548+0000 7f209bfff640 4 set_mon_vals no callback set 2024-03-25T19:39:16.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.549+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f209c019ae0 con 0x7f20a414be90 2024-03-25T19:39:16.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.550+0000 7f20a9cf9640 10 monclient: get_auth_request con 0x7f208806e0b0 auth_method 0 2024-03-25T19:39:16.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.550+0000 7f209a7fc640 10 client.4683.objecter ms_handle_connect 0x7f208806e0b0 2024-03-25T19:39:16.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.556+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1490778728 0 0) 0x7f209c053660 con 0x7f20a414be90 2024-03-25T19:39:16.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.556+0000 7f209a7fc640 10 client.4683.objecter ms_dispatch 0x7f20a4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:16.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.556+0000 7f209a7fc640 3 client.4683.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:16.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.556+0000 7f209a7fc640 3 client.4683.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:16.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.556+0000 7f209a7fc640 20 client.4683.objecter dump_active .. 0 homeless 2024-03-25T19:39:16.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.556+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f209c04efa0 con 0x7f20a414be90 2024-03-25T19:39:16.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.556+0000 7f209a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:16.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.556+0000 7f209a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:16.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.813+0000 7f20abf5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:16.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.813+0000 7f20abf5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:16.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.813+0000 7f20abf5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:16.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.813+0000 7f20abf5d640 1 -- 172.21.15.19:0/1441571020 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20a4142460 con 0x7f20a414be90 2024-03-25T19:39:16.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.843+0000 7f209a7fc640 1 -- 172.21.15.19:0/1441571020 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1576595288) 0x7f20a4142460 con 0x7f20a414be90 2024-03-25T19:39:16.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.844+0000 7f209a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:16.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.844+0000 7f209a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:16.844 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:39:16.844 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:38:58.470042+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:58.470042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:16.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 10 librados: watch_flush enter 2024-03-25T19:39:16.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 10 librados: watch_flush exit 2024-03-25T19:39:16.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 20 client.4683.objecter shutdown clearing up homeless session... 2024-03-25T19:39:16.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 10 client.4683.objecter successfully canceled tick 2024-03-25T19:39:16.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 1 -- 172.21.15.19:0/1441571020 >> v1:172.21.15.188:6800/2207506229 conn(0x7f208806e0b0 legacy=0x7f2088070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 10 monclient: shutdown 2024-03-25T19:39:16.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:16.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 1 -- 172.21.15.19:0/1441571020 >> v1:172.21.15.19:6789/0 conn(0x7f20a414be90 legacy=0x7f20a41db270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:16.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 1 -- 172.21.15.19:0/1441571020 shutdown_connections 2024-03-25T19:39:16.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 1 -- 172.21.15.19:0/1441571020 >> 172.21.15.19:0/1441571020 conn(0x7f20a40617a0 msgr2=0x7f20a4145170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:16.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 1 -- 172.21.15.19:0/1441571020 shutdown_connections 2024-03-25T19:39:16.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 1 -- 172.21.15.19:0/1441571020 wait complete. 2024-03-25T19:39:16.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:16.846+0000 7f20abf5d640 1 librados: shutdown 2024-03-25T19:39:16.858 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:18.610 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:41.340 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.662 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:41.392 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.712 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:41.442 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.733 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:41.462 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.740 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:10:41.443 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.741 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:41.471 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.745 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:10:41.448 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.784 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:10:41.487 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.798 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:41.527 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:23.663 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:23.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.781+0000 7fe5ec2ff640 10 monclient: get_monmap_and_config 2024-03-25T19:39:23.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.781+0000 7fe5ec2ff640 10 monclient: build_initial_monmap 2024-03-25T19:39:23.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.781+0000 7fe5ec2ff640 10 monclient: monmap: 2024-03-25T19:39:23.782 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:23.782 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:23.782 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:39:23.783004+0000 2024-03-25T19:39:23.782 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:39:23.783004+0000 2024-03-25T19:39:23.782 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:23.782 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:23.782 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:23.782 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:39:23.782 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:39:23.782 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:23.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.782+0000 7fe5ec2ff640 1 Processor -- start 2024-03-25T19:39:23.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.782+0000 7fe5ec2ff640 1 -- start start 2024-03-25T19:39:23.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.782+0000 7fe5ec2ff640 10 monclient: init 2024-03-25T19:39:23.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.783+0000 7fe5ec2ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:23.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.783+0000 7fe5ec2ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:23.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.783+0000 7fe5ec2ff640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e413fc60 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:23.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.783+0000 7fe5ec2ff640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e4070570 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:23.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.783+0000 7fe5ec2ff640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e40727a0 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:23.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.783+0000 7fe5ec2ff640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e413c9f0 con 0x7fe5e40727a0 2024-03-25T19:39:23.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.783+0000 7fe5ec2ff640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e406fac0 con 0x7fe5e413fc60 2024-03-25T19:39:23.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.783+0000 7fe5ec2ff640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e40715d0 con 0x7fe5e4070570 2024-03-25T19:39:23.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.783+0000 7fe5ec2ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:23.784 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.783+0000 7fe5ec2ff640 10 monclient(hunting): authenticate will time out at 6617.053499s 2024-03-25T19:39:23.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.786+0000 7fe5ea09b640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fe5e413fc60 0x7fe5e406f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:35466/0 (socket says 172.21.15.19:35466) 2024-03-25T19:39:23.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.786+0000 7fe5ea09b640 1 -- 172.21.15.19:0/3737134283 learned_addr learned my addr 172.21.15.19:0/3737134283 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.795+0000 7fe5e9099640 1 -- 172.21.15.19:0/3737134283 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205641641 0 0) 0x7fe5e406fac0 con 0x7fe5e413fc60 2024-03-25T19:39:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.795+0000 7fe5e9099640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.795+0000 7fe5e9099640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.795+0000 7fe5e9099640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.795+0000 7fe5e9099640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:39:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.795+0000 7fe5e9099640 1 -- 172.21.15.19:0/3737134283 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d0003520 con 0x7fe5e413fc60 2024-03-25T19:39:23.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 1 -- 172.21.15.19:0/3737134283 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050725037 0 0) 0x7fe5e40715d0 con 0x7fe5e4070570 2024-03-25T19:39:23.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 1 -- 172.21.15.19:0/3737134283 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e406fac0 con 0x7fe5e4070570 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 1 -- 172.21.15.19:0/3737134283 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1714156326 0 0) 0x7fe5d0003520 con 0x7fe5e413fc60 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 1 -- 172.21.15.19:0/3737134283 >> v1:172.21.15.19:6790/0 conn(0x7fe5e4070570 legacy=0x7fe5e4072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 1 -- 172.21.15.19:0/3737134283 >> v1:172.21.15.19:6789/0 conn(0x7fe5e40727a0 legacy=0x7fe5e414a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 10 monclient: _finish_hunting 0 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 1 monclient: found mon.noname-c 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 1 -- 172.21.15.19:0/3737134283 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e40718b0 con 0x7fe5e413fc60 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 10 monclient: _finish_auth 0 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 10 monclient: _check_auth_tickets 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 1 -- 172.21.15.19:0/3737134283 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe5e0001690 con 0x7fe5e413fc60 2024-03-25T19:39:23.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5e9099640 10 monclient: dump: 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5ec2ff640 5 monclient: authenticate success, global_id 4684 2024-03-25T19:39:23.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.801+0000 7fe5ec2ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:23.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.810+0000 7fe5e9099640 1 -- 172.21.15.19:0/3737134283 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5e00017f0 con 0x7fe5e413fc60 2024-03-25T19:39:23.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.810+0000 7fe5e9099640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:23.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.810+0000 7fe5ec2ff640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:23.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.810+0000 7fe5ec2ff640 4 set_mon_vals no callback set 2024-03-25T19:39:23.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.811+0000 7fe5e9099640 1 -- 172.21.15.19:0/3737134283 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe5e0001c00 con 0x7fe5e413fc60 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.811+0000 7fe5e9099640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.811+0000 7fe5e9099640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.811+0000 7fe5e9099640 10 monclient: dump: 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:23.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.811+0000 7fe5ec2ff640 10 monclient: shutdown 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.811+0000 7fe5ec2ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.811+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/3737134283 >> v1:172.21.15.188:6789/0 conn(0x7fe5e413fc60 legacy=0x7fe5e406f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.811+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/3737134283 shutdown_connections 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.811+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/3737134283 >> 172.21.15.19:0/3737134283 conn(0x7fe5e40b4a80 msgr2=0x7fe5e40b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.811+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/3737134283 shutdown_connections 2024-03-25T19:39:23.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.811+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/3737134283 wait complete. 2024-03-25T19:39:23.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.812+0000 7fe5ec2ff640 10 monclient: build_initial_monmap 2024-03-25T19:39:23.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.812+0000 7fe5ec2ff640 10 monclient: monmap: 2024-03-25T19:39:23.815 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:23.815 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:23.815 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:39:23.815 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:39:23.815 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:23.815 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:39:23.815 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:23.815 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:39:23.815 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.813+0000 7fe5ec2ff640 1 librados: starting msgr at 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.813+0000 7fe5ec2ff640 1 librados: starting objecter 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.813+0000 7fe5ec2ff640 1 Processor -- start 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.813+0000 7fe5ec2ff640 1 -- start start 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.813+0000 7fe5ec2ff640 1 librados: setting wanted keys 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.813+0000 7fe5ec2ff640 1 librados: calling monclient init 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.813+0000 7fe5ec2ff640 10 monclient: init 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.814+0000 7fe5ec2ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.814+0000 7fe5ec2ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.814+0000 7fe5ec2ff640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e4070570 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.814+0000 7fe5ec2ff640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e40727a0 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.814+0000 7fe5ec2ff640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e413fc60 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.814+0000 7fe5ec2ff640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e40ac8c0 con 0x7fe5e40727a0 2024-03-25T19:39:23.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.814+0000 7fe5ec2ff640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e41d8c10 con 0x7fe5e413fc60 2024-03-25T19:39:23.817 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.814+0000 7fe5ec2ff640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e41d9840 con 0x7fe5e4070570 2024-03-25T19:39:23.817 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.814+0000 7fe5ec2ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:23.817 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.814+0000 7fe5ec2ff640 10 monclient(hunting): authenticate will time out at 6617.084557s 2024-03-25T19:39:23.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.823+0000 7fe5e989a640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fe5e40727a0 0x7fe5e4136be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:56980/0 (socket says 172.21.15.19:56980) 2024-03-25T19:39:23.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.823+0000 7fe5e989a640 1 -- 172.21.15.19:0/2000978427 learned_addr learned my addr 172.21.15.19:0/2000978427 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:23.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.823+0000 7fe5daffd640 10 client.?.objecter ms_handle_connect 0x7fe5e40727a0 2024-03-25T19:39:23.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.823+0000 7fe5daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.828+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329034917 0 0) 0x7fe5e40ac8c0 con 0x7fe5e40727a0 2024-03-25T19:39:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.828+0000 7fe5daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.828+0000 7fe5daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.828+0000 7fe5daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.828+0000 7fe5daffd640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:39:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.828+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5c8004eb0 con 0x7fe5e40727a0 2024-03-25T19:39:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.828+0000 7fe5daffd640 10 client.?.objecter ms_handle_connect 0x7fe5e413fc60 2024-03-25T19:39:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.828+0000 7fe5daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:23.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.831+0000 7fe5daffd640 10 client.?.objecter ms_handle_connect 0x7fe5e4070570 2024-03-25T19:39:23.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.831+0000 7fe5daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:23.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.840+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3872554831 0 0) 0x7fe5c8004eb0 con 0x7fe5e40727a0 2024-03-25T19:39:23.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.840+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5e40ac8c0 con 0x7fe5e40727a0 2024-03-25T19:39:23.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.840+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe5d4001990 con 0x7fe5e40727a0 2024-03-25T19:39:23.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.840+0000 7fe5daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:23.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.840+0000 7fe5daffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:23.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.840+0000 7fe5daffd640 10 monclient(hunting): dump: 2024-03-25T19:39:23.840 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:23.840 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:23.840 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:23.841 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:23.841 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:23.841 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:23.841 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:23.841 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:23.841 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:23.841 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:23.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.840+0000 7fe5daffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:23.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.849+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220278528 0 0) 0x7fe5e41d9840 con 0x7fe5e4070570 2024-03-25T19:39:23.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.849+0000 7fe5daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.849+0000 7fe5daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.849+0000 7fe5daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.849+0000 7fe5daffd640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:39:23.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.849+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5c8004eb0 con 0x7fe5e4070570 2024-03-25T19:39:23.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.851+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507589885 0 0) 0x7fe5e41d8c10 con 0x7fe5e413fc60 2024-03-25T19:39:23.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.851+0000 7fe5daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:23.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.851+0000 7fe5daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:23.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.851+0000 7fe5daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:23.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.851+0000 7fe5daffd640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:39:23.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.851+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e41d9840 con 0x7fe5e413fc60 2024-03-25T19:39:23.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1206637776 0 0) 0x7fe5e40ac8c0 con 0x7fe5e40727a0 2024-03-25T19:39:23.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 >> v1:172.21.15.19:6790/0 conn(0x7fe5e4070570 legacy=0x7fe5e41d9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 >> v1:172.21.15.188:6789/0 conn(0x7fe5e413fc60 legacy=0x7fe5e40aaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:23.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5daffd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:23.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5daffd640 1 monclient: found mon.a 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5daffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e413a100 con 0x7fe5e40727a0 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5daffd640 10 monclient: _finish_auth 0 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5daffd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5ec2ff640 5 monclient: authenticate success, global_id 4689 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5ec2ff640 10 monclient: _renew_subs 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5ec2ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/2000978427 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e413a350 con 0x7fe5e40727a0 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5ec2ff640 10 client.4689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5ec2ff640 10 monclient: _renew_subs 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5ec2ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/2000978427 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e41374d0 con 0x7fe5e40727a0 2024-03-25T19:39:23.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.861+0000 7fe5ec2ff640 1 librados: init done 2024-03-25T19:39:23.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.862+0000 7fe5ec2ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:23.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.862+0000 7fe5ec2ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:23.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.862+0000 7fe5ec2ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:23.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:23.862+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/2000978427 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e41d8dc0 con 0x7fe5e40727a0 2024-03-25T19:39:24.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.185+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5d4003610 con 0x7fe5e40727a0 2024-03-25T19:39:24.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.185+0000 7fe5daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:24.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.185+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fe5d4003880 con 0x7fe5e40727a0 2024-03-25T19:39:24.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.185+0000 7fe5daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:24.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.185+0000 7fe5daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.185+0000 7fe5daffd640 10 monclient: dump: 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.185+0000 7fe5e8898640 4 set_mon_vals no callback set 2024-03-25T19:39:24.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.186+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fe5d4003aa0 con 0x7fe5e40727a0 2024-03-25T19:39:24.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.187+0000 7fe5ea09b640 10 monclient: get_auth_request con 0x7fe5c806dea0 auth_method 0 2024-03-25T19:39:24.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.187+0000 7fe5daffd640 10 client.4689.objecter ms_handle_connect 0x7fe5c806dea0 2024-03-25T19:39:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.194+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1490778728 0 0) 0x7fe5d4054610 con 0x7fe5e40727a0 2024-03-25T19:39:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.194+0000 7fe5daffd640 10 client.4689.objecter ms_dispatch 0x7fe5e4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.194+0000 7fe5daffd640 3 client.4689.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.194+0000 7fe5daffd640 3 client.4689.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.194+0000 7fe5daffd640 20 client.4689.objecter dump_active .. 0 homeless 2024-03-25T19:39:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.194+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5d4050120 con 0x7fe5e40727a0 2024-03-25T19:39:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.194+0000 7fe5daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:24.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.194+0000 7fe5daffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:24.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.418+0000 7fe5ec2ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:24.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.418+0000 7fe5ec2ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:24.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.418+0000 7fe5ec2ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:24.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.418+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/2000978427 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5e41d8a00 con 0x7fe5e40727a0 2024-03-25T19:39:24.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.457+0000 7fe5daffd640 1 -- 172.21.15.19:0/2000978427 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1576595288) 0x7fe5d4052050 con 0x7fe5e40727a0 2024-03-25T19:39:24.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.457+0000 7fe5daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:24.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.457+0000 7fe5daffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:24.457 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:39:24.458 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:38:58.470042+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:58.470042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:24.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.459+0000 7fe5ec2ff640 10 librados: watch_flush enter 2024-03-25T19:39:24.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.460+0000 7fe5ec2ff640 10 librados: watch_flush exit 2024-03-25T19:39:24.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.460+0000 7fe5ec2ff640 20 client.4689.objecter shutdown clearing up homeless session... 2024-03-25T19:39:24.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.460+0000 7fe5ec2ff640 10 client.4689.objecter successfully canceled tick 2024-03-25T19:39:24.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.460+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/2000978427 >> v1:172.21.15.188:6800/2207506229 conn(0x7fe5c806dea0 legacy=0x7fe5c8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:24.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.460+0000 7fe5ec2ff640 10 monclient: shutdown 2024-03-25T19:39:24.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.460+0000 7fe5ec2ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:24.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.460+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/2000978427 >> v1:172.21.15.19:6789/0 conn(0x7fe5e40727a0 legacy=0x7fe5e4136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:24.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.460+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/2000978427 shutdown_connections 2024-03-25T19:39:24.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.460+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/2000978427 >> 172.21.15.19:0/2000978427 conn(0x7fe5e40b4a80 msgr2=0x7fe5e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:24.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.460+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/2000978427 shutdown_connections 2024-03-25T19:39:24.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.460+0000 7fe5ec2ff640 1 -- 172.21.15.19:0/2000978427 wait complete. 2024-03-25T19:39:24.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:24.460+0000 7fe5ec2ff640 1 librados: shutdown 2024-03-25T19:39:24.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:31.278 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.396+0000 7f7ece723640 10 monclient: get_monmap_and_config 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.396+0000 7f7ece723640 10 monclient: build_initial_monmap 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.396+0000 7f7ece723640 10 monclient: monmap: 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:39:31.397277+0000 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:39:31.397277+0000 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:39:31.396 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:31.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 1 Processor -- start 2024-03-25T19:39:31.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 1 -- start start 2024-03-25T19:39:31.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 10 monclient: init 2024-03-25T19:39:31.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:31.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:39:31.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec80a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:31.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec813f710 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:31.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec81430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:31.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec8142520 con 0x7f7ec80a98f0 2024-03-25T19:39:31.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80b28a0 con 0x7f7ec813f710 2024-03-25T19:39:31.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80b2a30 con 0x7f7ec81430c0 2024-03-25T19:39:31.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:31.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.397+0000 7f7ece723640 10 monclient(hunting): authenticate will time out at 6624.667801s 2024-03-25T19:39:31.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.408+0000 7f7ec7fff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f7ec80a98f0 0x7f7ec80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:49078/0 (socket says 172.21.15.19:49078) 2024-03-25T19:39:31.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.408+0000 7f7ec7fff640 1 -- 172.21.15.19:0/236137364 learned_addr learned my addr 172.21.15.19:0/236137364 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:31.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.419+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157634520 0 0) 0x7f7ec80b2a30 con 0x7f7ec81430c0 2024-03-25T19:39:31.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.419+0000 7f7ec77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.419+0000 7f7ec77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.419+0000 7f7ec77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.419+0000 7f7ec77fe640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:39:31.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.419+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb4003520 con 0x7f7ec81430c0 2024-03-25T19:39:31.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.421+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226895898 0 0) 0x7f7ec8142520 con 0x7f7ec80a98f0 2024-03-25T19:39:31.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.421+0000 7f7ec77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.421+0000 7f7ec77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.421+0000 7f7ec77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.421+0000 7f7ec77fe640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:39:31.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.421+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec80b2a30 con 0x7f7ec80a98f0 2024-03-25T19:39:31.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.437+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247626889 0 0) 0x7f7ec80b28a0 con 0x7f7ec813f710 2024-03-25T19:39:31.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.437+0000 7f7ec77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.437+0000 7f7ec77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.437+0000 7f7ec77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.437+0000 7f7ec77fe640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:39:31.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.437+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec8142520 con 0x7f7ec813f710 2024-03-25T19:39:31.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2394497115 0 0) 0x7f7eb4003520 con 0x7f7ec81430c0 2024-03-25T19:39:31.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 >> v1:172.21.15.188:6789/0 conn(0x7f7ec813f710 legacy=0x7f7ec81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 >> v1:172.21.15.19:6789/0 conn(0x7f7ec80a98f0 legacy=0x7f7ec80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:31.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 1 monclient: found mon.noname-b 2024-03-25T19:39:31.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:31.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:39:31.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec80b44c0 con 0x7f7ec81430c0 2024-03-25T19:39:31.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:31.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:31.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:31.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7ec0001a10 con 0x7f7ec81430c0 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ec77fe640 10 monclient: dump: 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ece723640 5 monclient: authenticate success, global_id 4667 2024-03-25T19:39:31.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.449+0000 7f7ece723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:31.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.468+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ec0001bf0 con 0x7f7ec81430c0 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.468+0000 7f7ec77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.468+0000 7f7ec77fe640 1 -- 172.21.15.19:0/236137364 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7ec0002000 con 0x7f7ec81430c0 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.468+0000 7f7ec77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.469+0000 7f7ec77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.469+0000 7f7ec77fe640 10 monclient: dump: 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:31.469 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:31.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.469+0000 7f7ece723640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:31.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.469+0000 7f7ece723640 4 set_mon_vals no callback set 2024-03-25T19:39:31.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.469+0000 7f7ece723640 10 monclient: shutdown 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.469+0000 7f7ece723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.469+0000 7f7ece723640 1 -- 172.21.15.19:0/236137364 >> v1:172.21.15.19:6790/0 conn(0x7f7ec81430c0 legacy=0x7f7ec814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.470+0000 7f7ece723640 1 -- 172.21.15.19:0/236137364 shutdown_connections 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.470+0000 7f7ece723640 1 -- 172.21.15.19:0/236137364 >> 172.21.15.19:0/236137364 conn(0x7f7ec80b7ea0 msgr2=0x7f7ec80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.470+0000 7f7ece723640 1 -- 172.21.15.19:0/236137364 shutdown_connections 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.470+0000 7f7ece723640 1 -- 172.21.15.19:0/236137364 wait complete. 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.471+0000 7f7ece723640 10 monclient: build_initial_monmap 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.471+0000 7f7ece723640 10 monclient: monmap: 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:31.471 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:39:31.472 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:31.472 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:39:31.472 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:39:31.472 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:31.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.471+0000 7f7ece723640 1 librados: starting msgr at 2024-03-25T19:39:31.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.471+0000 7f7ece723640 1 librados: starting objecter 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.471+0000 7f7ece723640 1 Processor -- start 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.471+0000 7f7ece723640 1 -- start start 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.471+0000 7f7ece723640 1 librados: setting wanted keys 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.471+0000 7f7ece723640 1 librados: calling monclient init 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.471+0000 7f7ece723640 10 monclient: init 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.472+0000 7f7ece723640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.472+0000 7f7ece723640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.472+0000 7f7ece723640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec80a98f0 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.472+0000 7f7ece723640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.472+0000 7f7ece723640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec81430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.472+0000 7f7ece723640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec8136ad0 con 0x7f7ec813f710 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.472+0000 7f7ece723640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80b7520 con 0x7f7ec81430c0 2024-03-25T19:39:31.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.472+0000 7f7ece723640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec8135810 con 0x7f7ec80a98f0 2024-03-25T19:39:31.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.472+0000 7f7ece723640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:31.475 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.472+0000 7f7ece723640 10 monclient(hunting): authenticate will time out at 6624.742741s 2024-03-25T19:39:31.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.489+0000 7f7ec7fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f7ec80a98f0 0x7f7ec8135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:38088/0 (socket says 172.21.15.19:38088) 2024-03-25T19:39:31.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.489+0000 7f7ec7fff640 1 -- 172.21.15.19:0/3297041491 learned_addr learned my addr 172.21.15.19:0/3297041491 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:31.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.492+0000 7f7ec57fa640 10 client.?.objecter ms_handle_connect 0x7f7ec81430c0 2024-03-25T19:39:31.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.492+0000 7f7ec57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:31.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.496+0000 7f7ec57fa640 10 client.?.objecter ms_handle_connect 0x7f7ec80a98f0 2024-03-25T19:39:31.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.496+0000 7f7ec57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:31.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.502+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628913887 0 0) 0x7f7ec8135810 con 0x7f7ec80a98f0 2024-03-25T19:39:31.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.502+0000 7f7ec57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.502+0000 7f7ec57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.502+0000 7f7ec57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.502+0000 7f7ec57fa640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:39:31.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.502+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea8004eb0 con 0x7f7ec80a98f0 2024-03-25T19:39:31.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.502+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444067860 0 0) 0x7f7ec80b7520 con 0x7f7ec81430c0 2024-03-25T19:39:31.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.502+0000 7f7ec57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.502+0000 7f7ec57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.502+0000 7f7ec57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.502+0000 7f7ec57fa640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:39:31.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.502+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec8135810 con 0x7f7ec81430c0 2024-03-25T19:39:31.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.506+0000 7f7ec57fa640 10 client.?.objecter ms_handle_connect 0x7f7ec813f710 2024-03-25T19:39:31.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.506+0000 7f7ec57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:31.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.514+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627127924 0 0) 0x7f7ec8136ad0 con 0x7f7ec813f710 2024-03-25T19:39:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.514+0000 7f7ec57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:39:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.514+0000 7f7ec57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.514+0000 7f7ec57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.514+0000 7f7ec57fa640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:39:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.514+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec80b7520 con 0x7f7ec813f710 2024-03-25T19:39:31.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.521+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2771245245 0 0) 0x7f7ec8135810 con 0x7f7ec81430c0 2024-03-25T19:39:31.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.521+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ec8136ad0 con 0x7f7ec81430c0 2024-03-25T19:39:31.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.521+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7ec0001f00 con 0x7f7ec81430c0 2024-03-25T19:39:31.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.521+0000 7f7ec57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.521+0000 7f7ec57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:31.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.521+0000 7f7ec57fa640 10 monclient(hunting): dump: 2024-03-25T19:39:31.522 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:31.522 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:31.522 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:31.522 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:31.522 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.522 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:31.522 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:31.522 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:31.522 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:31.522 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:31.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.521+0000 7f7ec57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:31.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.523+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 956718470 0 0) 0x7f7ec80b7520 con 0x7f7ec813f710 2024-03-25T19:39:31.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.523+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ec8135810 con 0x7f7ec813f710 2024-03-25T19:39:31.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.523+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7eb0001830 con 0x7f7ec813f710 2024-03-25T19:39:31.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.523+0000 7f7ec57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.523+0000 7f7ec57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:31.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.523+0000 7f7ec57fa640 10 monclient(hunting): dump: 2024-03-25T19:39:31.523 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:31.524 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:31.524 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:31.524 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:31.524 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.524 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:31.524 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:31.524 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:31.524 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:31.524 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:31.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.536+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3604905280 0 0) 0x7f7ea8004eb0 con 0x7f7ec80a98f0 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.536+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ec80b7520 con 0x7f7ec80a98f0 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.536+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7eb8001970 con 0x7f7ec80a98f0 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.536+0000 7f7ec57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.536+0000 7f7ec57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.536+0000 7f7ec57fa640 10 monclient(hunting): dump: 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:31.537 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:31.538 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:31.538 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:31.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2680722628 0 0) 0x7f7ec8135810 con 0x7f7ec813f710 2024-03-25T19:39:31.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 >> v1:172.21.15.19:6790/0 conn(0x7f7ec80a98f0 legacy=0x7f7ec8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 >> v1:172.21.15.188:6789/0 conn(0x7f7ec81430c0 legacy=0x7f7ec8130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:31.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ec57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:39:31.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ec57fa640 1 monclient: found mon.a 2024-03-25T19:39:31.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ec57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:31.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ec57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:31.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec80b6180 con 0x7f7ec813f710 2024-03-25T19:39:31.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ec57fa640 10 monclient: _finish_auth 0 2024-03-25T19:39:31.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ec57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:39:31.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ec57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:31.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ece723640 5 monclient: authenticate success, global_id 4695 2024-03-25T19:39:31.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ece723640 10 monclient: _renew_subs 2024-03-25T19:39:31.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ece723640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:31.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ece723640 1 -- 172.21.15.19:0/3297041491 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec80b7a40 con 0x7f7ec813f710 2024-03-25T19:39:31.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ece723640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:31.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ece723640 10 monclient: _renew_subs 2024-03-25T19:39:31.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ece723640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:31.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ece723640 1 -- 172.21.15.19:0/3297041491 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec8131840 con 0x7f7ec813f710 2024-03-25T19:39:31.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.541+0000 7f7ece723640 1 librados: init done 2024-03-25T19:39:31.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.542+0000 7f7ece723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:31.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.542+0000 7f7ece723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:31.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.542+0000 7f7ece723640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:31.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.542+0000 7f7ece723640 1 -- 172.21.15.19:0/3297041491 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec8136f60 con 0x7f7ec813f710 2024-03-25T19:39:31.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.844+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7eb0001d20 con 0x7f7ec813f710 2024-03-25T19:39:31.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.844+0000 7f7ec57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:31.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.844+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f7eb0003610 con 0x7f7ec813f710 2024-03-25T19:39:31.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.844+0000 7f7ec57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:31.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.844+0000 7f7ec57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:31.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.844+0000 7f7ec57fa640 10 monclient: dump: 2024-03-25T19:39:31.845 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:31.845 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:31.845 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:31.845 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:31.845 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:31.845 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:31.845 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:31.845 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:31.845 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:31.845 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:31.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.844+0000 7f7ec6ffd640 4 set_mon_vals no callback set 2024-03-25T19:39:31.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.845+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f7eb0019660 con 0x7f7ec813f710 2024-03-25T19:39:31.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.845+0000 7f7ec7fff640 10 monclient: get_auth_request con 0x7f7ea806e0c0 auth_method 0 2024-03-25T19:39:31.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.846+0000 7f7ec57fa640 10 client.4695.objecter ms_handle_connect 0x7f7ea806e0c0 2024-03-25T19:39:31.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.851+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1490778728 0 0) 0x7f7eb0053380 con 0x7f7ec813f710 2024-03-25T19:39:31.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.851+0000 7f7ec57fa640 10 client.4695.objecter ms_dispatch 0x7f7ec8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:31.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.851+0000 7f7ec57fa640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:31.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.851+0000 7f7ec57fa640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:31.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.851+0000 7f7ec57fa640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T19:39:31.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.851+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7eb004ecc0 con 0x7f7ec813f710 2024-03-25T19:39:31.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.851+0000 7f7ec57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:31.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:31.851+0000 7f7ec57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:32.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.102+0000 7f7ece723640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:32.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.102+0000 7f7ece723640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:32.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.102+0000 7f7ece723640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:32.102 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.102+0000 7f7ece723640 1 -- 172.21.15.19:0/3297041491 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ec80b6310 con 0x7f7ec813f710 2024-03-25T19:39:32.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.195+0000 7f7ec57fa640 1 -- 172.21.15.19:0/3297041491 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1576595288) 0x7f7eb0050dc0 con 0x7f7ec813f710 2024-03-25T19:39:32.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.195+0000 7f7ec57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:32.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.195+0000 7f7ec57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:32.196 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:39:32.196 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:38:58.470042+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:58.470042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:32.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 10 librados: watch_flush enter 2024-03-25T19:39:32.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 10 librados: watch_flush exit 2024-03-25T19:39:32.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T19:39:32.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 10 client.4695.objecter successfully canceled tick 2024-03-25T19:39:32.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 1 -- 172.21.15.19:0/3297041491 >> v1:172.21.15.188:6800/2207506229 conn(0x7f7ea806e0c0 legacy=0x7f7ea8070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:32.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 10 monclient: shutdown 2024-03-25T19:39:32.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:32.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 1 -- 172.21.15.19:0/3297041491 >> v1:172.21.15.19:6789/0 conn(0x7f7ec813f710 legacy=0x7f7ec80b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:32.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 1 -- 172.21.15.19:0/3297041491 shutdown_connections 2024-03-25T19:39:32.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 1 -- 172.21.15.19:0/3297041491 >> 172.21.15.19:0/3297041491 conn(0x7f7ec80b7ea0 msgr2=0x7f7ec8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:32.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 1 -- 172.21.15.19:0/3297041491 shutdown_connections 2024-03-25T19:39:32.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 1 -- 172.21.15.19:0/3297041491 wait complete. 2024-03-25T19:39:32.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:32.198+0000 7f7ece723640 1 librados: shutdown 2024-03-25T19:39:32.208 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:38.724 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:01.454 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.739 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:01.468 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.759 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:01.488 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.774 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:01.503 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.777 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:01.507 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.809 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:11:01.512 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.816 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:01.545 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.816 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:11:01.520 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.846 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:11:01.549 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:39.013 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:39.128 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.128+0000 7f1c525f6640 10 monclient: get_monmap_and_config 2024-03-25T19:39:39.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.128+0000 7f1c525f6640 10 monclient: build_initial_monmap 2024-03-25T19:39:39.129 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.129+0000 7f1c525f6640 10 monclient: monmap: 2024-03-25T19:39:39.129 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:39.129 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:39.129 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:39:39.130201+0000 2024-03-25T19:39:39.129 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:39:39.130201+0000 2024-03-25T19:39:39.129 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:39.129 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:39.129 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:39:39.130 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.129+0000 7f1c525f6640 1 Processor -- start 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.129+0000 7f1c525f6640 1 -- start start 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.129+0000 7f1c525f6640 10 monclient: init 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.130+0000 7f1c525f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:39.131 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.130+0000 7f1c525f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:39.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.130+0000 7f1c525f6640 10 monclient(hunting): picked mon.noname-c con 0x7f1c4c061890 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:39.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.130+0000 7f1c525f6640 10 monclient(hunting): picked mon.noname-b con 0x7f1c4c06e740 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:39.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.130+0000 7f1c525f6640 10 monclient(hunting): picked mon.noname-a con 0x7f1c4c0ae280 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:39.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.130+0000 7f1c525f6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c0ae830 con 0x7f1c4c0ae280 2024-03-25T19:39:39.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.130+0000 7f1c525f6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c06f720 con 0x7f1c4c061890 2024-03-25T19:39:39.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.130+0000 7f1c525f6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c0ada80 con 0x7f1c4c06e740 2024-03-25T19:39:39.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.130+0000 7f1c525f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:39.132 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.130+0000 7f1c525f6640 10 monclient(hunting): authenticate will time out at 6632.400826s 2024-03-25T19:39:39.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.138+0000 7f1c50b93640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f1c4c0ae280 0x7f1c4c145f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:53662/0 (socket says 172.21.15.19:53662) 2024-03-25T19:39:39.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.138+0000 7f1c50b93640 1 -- 172.21.15.19:0/650439285 learned_addr learned my addr 172.21.15.19:0/650439285 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:39.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.149+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935148375 0 0) 0x7f1c4c0ae830 con 0x7f1c4c0ae280 2024-03-25T19:39:39.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.149+0000 7f1c4affd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:39.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.149+0000 7f1c4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:39.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.149+0000 7f1c4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:39.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.149+0000 7f1c4affd640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:39:39.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.149+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c34003520 con 0x7f1c4c0ae280 2024-03-25T19:39:39.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.157+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891468677 0 0) 0x7f1c4c06f720 con 0x7f1c4c061890 2024-03-25T19:39:39.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.157+0000 7f1c4affd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:39.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.157+0000 7f1c4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:39.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.157+0000 7f1c4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:39.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.157+0000 7f1c4affd640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:39:39.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.157+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c4c0ae830 con 0x7f1c4c061890 2024-03-25T19:39:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.163+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609331641 0 0) 0x7f1c4c0ada80 con 0x7f1c4c06e740 2024-03-25T19:39:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.163+0000 7f1c4affd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.163+0000 7f1c4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.163+0000 7f1c4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.163+0000 7f1c4affd640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:39:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.163+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c4c06f720 con 0x7f1c4c06e740 2024-03-25T19:39:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 278479640 0 0) 0x7f1c34003520 con 0x7f1c4c0ae280 2024-03-25T19:39:39.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 >> v1:172.21.15.19:6790/0 conn(0x7f1c4c06e740 legacy=0x7f1c4c145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 >> v1:172.21.15.188:6789/0 conn(0x7f1c4c061890 legacy=0x7f1c4c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 1 monclient: found mon.noname-a 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c13fbd0 con 0x7f1c4c0ae280 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 10 monclient: _finish_auth 0 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1c44001a10 con 0x7f1c4c0ae280 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c525f6640 5 monclient: authenticate success, global_id 4698 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c525f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c4affd640 10 monclient: dump: 2024-03-25T19:39:39.165 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:39.166 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:39.166 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:39.166 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:39.166 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.166 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:39.166 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:39.166 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:39.166 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:39.166 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:39.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.164+0000 7f1c525f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:39.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.172+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c44001bf0 con 0x7f1c4c0ae280 2024-03-25T19:39:39.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.172+0000 7f1c4affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.172+0000 7f1c4affd640 1 -- 172.21.15.19:0/650439285 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1c44002000 con 0x7f1c4c0ae280 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.172+0000 7f1c525f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.172+0000 7f1c525f6640 4 set_mon_vals no callback set 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.172+0000 7f1c4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.172+0000 7f1c4affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.172+0000 7f1c4affd640 10 monclient: dump: 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:39.173 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:39.174 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.173+0000 7f1c525f6640 10 monclient: shutdown 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.173+0000 7f1c525f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.173+0000 7f1c525f6640 1 -- 172.21.15.19:0/650439285 >> v1:172.21.15.19:6789/0 conn(0x7f1c4c0ae280 legacy=0x7f1c4c145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.173+0000 7f1c525f6640 1 -- 172.21.15.19:0/650439285 shutdown_connections 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.173+0000 7f1c525f6640 1 -- 172.21.15.19:0/650439285 >> 172.21.15.19:0/650439285 conn(0x7f1c4c0b0ef0 msgr2=0x7f1c4c0b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.173+0000 7f1c525f6640 1 -- 172.21.15.19:0/650439285 shutdown_connections 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.173+0000 7f1c525f6640 1 -- 172.21.15.19:0/650439285 wait complete. 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.174+0000 7f1c525f6640 10 monclient: build_initial_monmap 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.174+0000 7f1c525f6640 10 monclient: monmap: 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:39:39.176 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.174+0000 7f1c525f6640 1 librados: starting msgr at 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.174+0000 7f1c525f6640 1 librados: starting objecter 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.174+0000 7f1c525f6640 1 Processor -- start 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.174+0000 7f1c525f6640 1 -- start start 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.174+0000 7f1c525f6640 1 librados: setting wanted keys 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.174+0000 7f1c525f6640 1 librados: calling monclient init 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.174+0000 7f1c525f6640 10 monclient: init 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.175+0000 7f1c525f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.175+0000 7f1c525f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:39.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.175+0000 7f1c525f6640 10 monclient(hunting): picked mon.noname-c con 0x7f1c4c061890 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:39.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.175+0000 7f1c525f6640 10 monclient(hunting): picked mon.noname-a con 0x7f1c4c0ae280 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:39.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.175+0000 7f1c525f6640 10 monclient(hunting): picked mon.noname-b con 0x7f1c4c143200 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:39.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.175+0000 7f1c525f6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c1414e0 con 0x7f1c4c0ae280 2024-03-25T19:39:39.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.175+0000 7f1c525f6640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c1400a0 con 0x7f1c4c143200 2024-03-25T19:39:39.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.175+0000 7f1c525f6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4c140230 con 0x7f1c4c061890 2024-03-25T19:39:39.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.175+0000 7f1c525f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:39.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.175+0000 7f1c525f6640 10 monclient(hunting): authenticate will time out at 6632.445627s 2024-03-25T19:39:39.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.187+0000 7f1c4b7fe640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f1c4c0ae280 0x7f1c4c1d7640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:53694/0 (socket says 172.21.15.19:53694) 2024-03-25T19:39:39.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.187+0000 7f1c4b7fe640 1 -- 172.21.15.19:0/4026585832 learned_addr learned my addr 172.21.15.19:0/4026585832 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:39.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.188+0000 7f1c48ff9640 10 client.?.objecter ms_handle_connect 0x7f1c4c0ae280 2024-03-25T19:39:39.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.188+0000 7f1c48ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:39.192 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.192+0000 7f1c48ff9640 10 client.?.objecter ms_handle_connect 0x7f1c4c143200 2024-03-25T19:39:39.192 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.192+0000 7f1c48ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:39.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.195+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310913308 0 0) 0x7f1c4c1414e0 con 0x7f1c4c0ae280 2024-03-25T19:39:39.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.195+0000 7f1c48ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:39.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.195+0000 7f1c48ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:39.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.195+0000 7f1c48ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:39.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.195+0000 7f1c48ff9640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:39:39.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.195+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c38004eb0 con 0x7f1c4c0ae280 2024-03-25T19:39:39.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.204+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 744213518 0 0) 0x7f1c38004eb0 con 0x7f1c4c0ae280 2024-03-25T19:39:39.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.204+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c4c1414e0 con 0x7f1c4c0ae280 2024-03-25T19:39:39.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.204+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1c40001a10 con 0x7f1c4c0ae280 2024-03-25T19:39:39.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.204+0000 7f1c48ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.204+0000 7f1c48ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:39.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.204+0000 7f1c48ff9640 10 monclient(hunting): dump: 2024-03-25T19:39:39.204 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:39.204 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:39.204 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:39.205 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:39.205 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.205 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:39.205 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:39.205 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:39.205 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:39.205 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:39.205 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.204+0000 7f1c48ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:39.205 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.206+0000 7f1c48ff9640 10 client.?.objecter ms_handle_connect 0x7f1c4c061890 2024-03-25T19:39:39.205 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.206+0000 7f1c48ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:39.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.208+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344529953 0 0) 0x7f1c4c1400a0 con 0x7f1c4c143200 2024-03-25T19:39:39.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.208+0000 7f1c48ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:39:39.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.208+0000 7f1c48ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:39.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.208+0000 7f1c48ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:39.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.208+0000 7f1c48ff9640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:39:39.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.208+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c38004eb0 con 0x7f1c4c143200 2024-03-25T19:39:39.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4052243002 0 0) 0x7f1c4c1414e0 con 0x7f1c4c0ae280 2024-03-25T19:39:39.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 >> v1:172.21.15.19:6790/0 conn(0x7f1c4c061890 legacy=0x7f1c4c146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 >> v1:172.21.15.188:6789/0 conn(0x7f1c4c143200 legacy=0x7f1c4c1d7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c48ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c48ff9640 1 monclient: found mon.a 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c48ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c48ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c1e5f10 con 0x7f1c4c0ae280 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c48ff9640 10 monclient: _finish_auth 0 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c48ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c525f6640 5 monclient: authenticate success, global_id 4701 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c525f6640 10 monclient: _renew_subs 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c525f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c525f6640 1 -- 172.21.15.19:0/4026585832 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4c1e2ee0 con 0x7f1c4c0ae280 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c525f6640 10 client.4701.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c525f6640 10 monclient: _renew_subs 2024-03-25T19:39:39.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c525f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:39.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c525f6640 1 -- 172.21.15.19:0/4026585832 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4c1e3460 con 0x7f1c4c0ae280 2024-03-25T19:39:39.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.219+0000 7f1c525f6640 1 librados: init done 2024-03-25T19:39:39.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.220+0000 7f1c525f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:39.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.220+0000 7f1c525f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:39.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.220+0000 7f1c525f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:39.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.220+0000 7f1c525f6640 1 -- 172.21.15.19:0/4026585832 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c4c0627b0 con 0x7f1c4c0ae280 2024-03-25T19:39:39.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.532+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c40001cf0 con 0x7f1c4c0ae280 2024-03-25T19:39:39.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.532+0000 7f1c48ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:39.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.532+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f1c40003a60 con 0x7f1c4c0ae280 2024-03-25T19:39:39.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.532+0000 7f1c48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:39.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.532+0000 7f1c48ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:39.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.532+0000 7f1c48ff9640 10 monclient: dump: 2024-03-25T19:39:39.533 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:39.533 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:39.533 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:39.533 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:39.533 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:39.533 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:39.533 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:39.533 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:39.533 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:39.533 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:39.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.532+0000 7f1c4a7fc640 4 set_mon_vals no callback set 2024-03-25T19:39:39.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.533+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f1c40003c80 con 0x7f1c4c0ae280 2024-03-25T19:39:39.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.533+0000 7f1c4bfff640 10 monclient: get_auth_request con 0x7f1c3806dd90 auth_method 0 2024-03-25T19:39:39.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.534+0000 7f1c48ff9640 10 client.4701.objecter ms_handle_connect 0x7f1c3806dd90 2024-03-25T19:39:39.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.539+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1490778728 0 0) 0x7f1c40054870 con 0x7f1c4c0ae280 2024-03-25T19:39:39.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.539+0000 7f1c48ff9640 10 client.4701.objecter ms_dispatch 0x7f1c4c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:39.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.539+0000 7f1c48ff9640 3 client.4701.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:39.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.539+0000 7f1c48ff9640 3 client.4701.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:39.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.539+0000 7f1c48ff9640 20 client.4701.objecter dump_active .. 0 homeless 2024-03-25T19:39:39.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.539+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c40050380 con 0x7f1c4c0ae280 2024-03-25T19:39:39.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.539+0000 7f1c48ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:39.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.539+0000 7f1c48ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:39.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.798+0000 7f1c525f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:39.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.798+0000 7f1c525f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:39.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.798+0000 7f1c525f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:39.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.798+0000 7f1c525f6640 1 -- 172.21.15.19:0/4026585832 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c4c069f40 con 0x7f1c4c0ae280 2024-03-25T19:39:39.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.827+0000 7f1c48ff9640 1 -- 172.21.15.19:0/4026585832 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1576595288) 0x7f1c400522b0 con 0x7f1c4c0ae280 2024-03-25T19:39:39.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.827+0000 7f1c48ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:39.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.827+0000 7f1c48ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:39.827 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:39:39.827 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:38:58.470042+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:58.470042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:39.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 10 librados: watch_flush enter 2024-03-25T19:39:39.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 10 librados: watch_flush exit 2024-03-25T19:39:39.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 20 client.4701.objecter shutdown clearing up homeless session... 2024-03-25T19:39:39.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 10 client.4701.objecter successfully canceled tick 2024-03-25T19:39:39.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 1 -- 172.21.15.19:0/4026585832 >> v1:172.21.15.188:6800/2207506229 conn(0x7f1c3806dd90 legacy=0x7f1c38070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 10 monclient: shutdown 2024-03-25T19:39:39.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:39.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 1 -- 172.21.15.19:0/4026585832 >> v1:172.21.15.19:6789/0 conn(0x7f1c4c0ae280 legacy=0x7f1c4c1d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:39.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 1 -- 172.21.15.19:0/4026585832 shutdown_connections 2024-03-25T19:39:39.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 1 -- 172.21.15.19:0/4026585832 >> 172.21.15.19:0/4026585832 conn(0x7f1c4c0b0ef0 msgr2=0x7f1c4c06fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:39.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 1 -- 172.21.15.19:0/4026585832 shutdown_connections 2024-03-25T19:39:39.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 1 -- 172.21.15.19:0/4026585832 wait complete. 2024-03-25T19:39:39.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:39.830+0000 7f1c525f6640 1 librados: shutdown 2024-03-25T19:39:39.841 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:46.648 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:46.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.765+0000 7f54276ae640 10 monclient: get_monmap_and_config 2024-03-25T19:39:46.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.765+0000 7f54276ae640 10 monclient: build_initial_monmap 2024-03-25T19:39:46.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.765+0000 7f54276ae640 10 monclient: monmap: 2024-03-25T19:39:46.766 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:46.766 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:46.766 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:39:46.767052+0000 2024-03-25T19:39:46.766 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:39:46.767052+0000 2024-03-25T19:39:46.766 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:46.766 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:46.766 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:46.766 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:39:46.766 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:39:46.767 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.766+0000 7f54276ae640 1 Processor -- start 2024-03-25T19:39:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.766+0000 7f54276ae640 1 -- start start 2024-03-25T19:39:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.766+0000 7f54276ae640 10 monclient: init 2024-03-25T19:39:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.767+0000 7f54276ae640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.767+0000 7f54276ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.767+0000 7f54276ae640 10 monclient(hunting): picked mon.noname-c con 0x7f54200a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.767+0000 7f54276ae640 10 monclient(hunting): picked mon.noname-a con 0x7f542013f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.767+0000 7f54276ae640 10 monclient(hunting): picked mon.noname-b con 0x7f54201430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.767+0000 7f54276ae640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5420142520 con 0x7f542013f710 2024-03-25T19:39:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.767+0000 7f54276ae640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54200b28a0 con 0x7f54200a98f0 2024-03-25T19:39:46.768 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.767+0000 7f54276ae640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54200b2a30 con 0x7f54201430c0 2024-03-25T19:39:46.769 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.767+0000 7f54276ae640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:46.769 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.767+0000 7f54276ae640 10 monclient(hunting): authenticate will time out at 6640.037294s 2024-03-25T19:39:46.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.773+0000 7f542544a640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f54200a98f0 0x7f54200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:41878/0 (socket says 172.21.15.19:41878) 2024-03-25T19:39:46.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.773+0000 7f542544a640 1 -- 172.21.15.19:0/2810417511 learned_addr learned my addr 172.21.15.19:0/2810417511 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:46.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.785+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759860897 0 0) 0x7f54200b28a0 con 0x7f54200a98f0 2024-03-25T19:39:46.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.785+0000 7f5417fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.785+0000 7f5417fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.785+0000 7f5417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.785 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.785+0000 7f5417fff640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:39:46.786 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.785+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f540c003520 con 0x7f54200a98f0 2024-03-25T19:39:46.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.804+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707250128 0 0) 0x7f5420142520 con 0x7f542013f710 2024-03-25T19:39:46.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.804+0000 7f5417fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.804+0000 7f5417fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.804+0000 7f5417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.804+0000 7f5417fff640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:39:46.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.804+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54200b28a0 con 0x7f542013f710 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.810+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912009930 0 0) 0x7f54200b2a30 con 0x7f54201430c0 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.810+0000 7f5417fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.810+0000 7f5417fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.810+0000 7f5417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.810+0000 7f5417fff640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:39:46.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.810+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5420142520 con 0x7f54201430c0 2024-03-25T19:39:46.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4144187675 0 0) 0x7f54200b28a0 con 0x7f542013f710 2024-03-25T19:39:46.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 >> v1:172.21.15.19:6790/0 conn(0x7f54201430c0 legacy=0x7f542014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 >> v1:172.21.15.188:6789/0 conn(0x7f54200a98f0 legacy=0x7f54200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 10 monclient: _finish_hunting 0 2024-03-25T19:39:46.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 1 monclient: found mon.noname-a 2024-03-25T19:39:46.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:46.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:39:46.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54200b44c0 con 0x7f542013f710 2024-03-25T19:39:46.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 10 monclient: _finish_auth 0 2024-03-25T19:39:46.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 10 monclient: _check_auth_tickets 2024-03-25T19:39:46.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:46.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5410001670 con 0x7f542013f710 2024-03-25T19:39:46.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:46.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:46.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f5417fff640 10 monclient: dump: 2024-03-25T19:39:46.814 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:46.814 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:46.814 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:46.814 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:46.815 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:46.815 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:46.815 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:46.815 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:46.815 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:46.815 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:46.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f54276ae640 5 monclient: authenticate success, global_id 4704 2024-03-25T19:39:46.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.812+0000 7f54276ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:46.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.835+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54100017d0 con 0x7f542013f710 2024-03-25T19:39:46.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.835+0000 7f5417fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:46.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.835+0000 7f5417fff640 1 -- 172.21.15.19:0/2810417511 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5410001be0 con 0x7f542013f710 2024-03-25T19:39:46.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.835+0000 7f5417fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.835+0000 7f5417fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.835+0000 7f5417fff640 10 monclient: dump: 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.835+0000 7f54276ae640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:46.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.835+0000 7f54276ae640 4 set_mon_vals no callback set 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.836+0000 7f54276ae640 10 monclient: shutdown 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.836+0000 7f54276ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.836+0000 7f54276ae640 1 -- 172.21.15.19:0/2810417511 >> v1:172.21.15.19:6789/0 conn(0x7f542013f710 legacy=0x7f54201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.836+0000 7f54276ae640 1 -- 172.21.15.19:0/2810417511 shutdown_connections 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.836+0000 7f54276ae640 1 -- 172.21.15.19:0/2810417511 >> 172.21.15.19:0/2810417511 conn(0x7f54200b7ea0 msgr2=0x7f54200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.836+0000 7f54276ae640 1 -- 172.21.15.19:0/2810417511 shutdown_connections 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.836+0000 7f54276ae640 1 -- 172.21.15.19:0/2810417511 wait complete. 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.837+0000 7f54276ae640 10 monclient: build_initial_monmap 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.837+0000 7f54276ae640 10 monclient: monmap: 2024-03-25T19:39:46.839 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.837+0000 7f54276ae640 1 librados: starting msgr at 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.837+0000 7f54276ae640 1 librados: starting objecter 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 1 Processor -- start 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 1 -- start start 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 1 librados: setting wanted keys 2024-03-25T19:39:46.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 1 librados: calling monclient init 2024-03-25T19:39:46.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 10 monclient: init 2024-03-25T19:39:46.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:46.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:46.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 10 monclient(hunting): picked mon.noname-b con 0x7f54200a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:46.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 10 monclient(hunting): picked mon.noname-a con 0x7f542013f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:46.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 10 monclient(hunting): picked mon.noname-c con 0x7f54201430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:46.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54200ab9e0 con 0x7f542013f710 2024-03-25T19:39:46.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54201d8360 con 0x7f54200a98f0 2024-03-25T19:39:46.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54201d5000 con 0x7f54201430c0 2024-03-25T19:39:46.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:46.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.838+0000 7f54276ae640 10 monclient(hunting): authenticate will time out at 6640.108566s 2024-03-25T19:39:46.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.845+0000 7f5424c49640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f542013f710 0x7f54201d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:51874/0 (socket says 172.21.15.19:51874) 2024-03-25T19:39:46.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.845+0000 7f5424c49640 1 -- 172.21.15.19:0/838608901 learned_addr learned my addr 172.21.15.19:0/838608901 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:46.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.846+0000 7f5415ffb640 10 client.?.objecter ms_handle_connect 0x7f542013f710 2024-03-25T19:39:46.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.846+0000 7f5415ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:46.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.848+0000 7f5415ffb640 10 client.?.objecter ms_handle_connect 0x7f54201430c0 2024-03-25T19:39:46.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.848+0000 7f5415ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:46.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.852+0000 7f5415ffb640 10 client.?.objecter ms_handle_connect 0x7f54200a98f0 2024-03-25T19:39:46.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.852+0000 7f5415ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:46.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.854+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497856346 0 0) 0x7f54200ab9e0 con 0x7f542013f710 2024-03-25T19:39:46.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.854+0000 7f5415ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.854+0000 7f5415ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.854+0000 7f5415ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.854+0000 7f5415ffb640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:39:46.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.854+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5404004eb0 con 0x7f542013f710 2024-03-25T19:39:46.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.864+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983918071 0 0) 0x7f54201d8360 con 0x7f54200a98f0 2024-03-25T19:39:46.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.864+0000 7f5415ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.864+0000 7f5415ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.864+0000 7f5415ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.864+0000 7f5415ffb640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:39:46.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.864+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54200ab9e0 con 0x7f54200a98f0 2024-03-25T19:39:46.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.868+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220333238 0 0) 0x7f54201d5000 con 0x7f54201430c0 2024-03-25T19:39:46.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.868+0000 7f5415ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:46.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.868+0000 7f5415ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:46.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.868+0000 7f5415ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:46.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.868+0000 7f5415ffb640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:39:46.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.868+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54201d8360 con 0x7f54201430c0 2024-03-25T19:39:46.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.870+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872297316 0 0) 0x7f5404004eb0 con 0x7f542013f710 2024-03-25T19:39:46.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.870+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54201d5000 con 0x7f542013f710 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.870+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5410001670 con 0x7f542013f710 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.870+0000 7f5415ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.870+0000 7f5415ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.870+0000 7f5415ffb640 10 monclient(hunting): dump: 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:46.871 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:46.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.870+0000 7f5415ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:46.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.875+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1994816339 0 0) 0x7f54200ab9e0 con 0x7f54200a98f0 2024-03-25T19:39:46.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.875+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5404004eb0 con 0x7f54200a98f0 2024-03-25T19:39:46.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.875+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f541c001950 con 0x7f54200a98f0 2024-03-25T19:39:46.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.875+0000 7f5415ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:46.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.875+0000 7f5415ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:46.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.875+0000 7f5415ffb640 10 monclient(hunting): dump: 2024-03-25T19:39:46.875 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:46.875 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:46.875 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:46.876 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:46.876 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:46.876 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:46.876 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:46.876 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:46.876 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:46.876 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:46.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 577025116 0 0) 0x7f54201d5000 con 0x7f542013f710 2024-03-25T19:39:46.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 >> v1:172.21.15.19:6790/0 conn(0x7f54201430c0 legacy=0x7f54201d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 >> v1:172.21.15.188:6789/0 conn(0x7f54200a98f0 legacy=0x7f5420138430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:46.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f5415ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:39:46.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f5415ffb640 1 monclient: found mon.a 2024-03-25T19:39:46.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f5415ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:46.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f5415ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:46.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54201d5190 con 0x7f542013f710 2024-03-25T19:39:46.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f5415ffb640 10 monclient: _finish_auth 0 2024-03-25T19:39:46.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f5415ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:39:46.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f5415ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:46.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f54276ae640 5 monclient: authenticate success, global_id 4707 2024-03-25T19:39:46.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f54276ae640 10 monclient: _renew_subs 2024-03-25T19:39:46.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f54276ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:46.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f54276ae640 1 -- 172.21.15.19:0/838608901 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54201d41e0 con 0x7f542013f710 2024-03-25T19:39:46.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f54276ae640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:46.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f54276ae640 10 monclient: _renew_subs 2024-03-25T19:39:46.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f54276ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:46.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f54276ae640 1 -- 172.21.15.19:0/838608901 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54201d4530 con 0x7f542013f710 2024-03-25T19:39:46.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.883+0000 7f54276ae640 1 librados: init done 2024-03-25T19:39:46.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.885+0000 7f54276ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:46.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.885+0000 7f54276ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:46.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.885+0000 7f54276ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:46.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:46.885+0000 7f54276ae640 1 -- 172.21.15.19:0/838608901 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54201d7590 con 0x7f542013f710 2024-03-25T19:39:47.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.206+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54100038a0 con 0x7f542013f710 2024-03-25T19:39:47.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.206+0000 7f5415ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:47.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.207+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f5410003b50 con 0x7f542013f710 2024-03-25T19:39:47.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.207+0000 7f5415ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:47.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.207+0000 7f5415ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:47.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.207+0000 7f5415ffb640 10 monclient: dump: 2024-03-25T19:39:47.207 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:47.207 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:47.207 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:47.207 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:47.208 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:47.208 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:47.208 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:47.208 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:47.208 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:47.208 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:47.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.207+0000 7f54177fe640 4 set_mon_vals no callback set 2024-03-25T19:39:47.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.207+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f5410003d70 con 0x7f542013f710 2024-03-25T19:39:47.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.208+0000 7f542544a640 10 monclient: get_auth_request con 0x7f540406df70 auth_method 0 2024-03-25T19:39:47.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.208+0000 7f5415ffb640 10 client.4707.objecter ms_handle_connect 0x7f540406df70 2024-03-25T19:39:47.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.214+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1490778728 0 0) 0x7f54100547b0 con 0x7f542013f710 2024-03-25T19:39:47.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.214+0000 7f5415ffb640 10 client.4707.objecter ms_dispatch 0x7f5420000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:47.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.214+0000 7f5415ffb640 3 client.4707.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:47.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.214+0000 7f5415ffb640 3 client.4707.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:47.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.214+0000 7f5415ffb640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T19:39:47.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.214+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54100502c0 con 0x7f542013f710 2024-03-25T19:39:47.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.214+0000 7f5415ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:47.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.214+0000 7f5415ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:47.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.447+0000 7f54276ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:47.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.447+0000 7f54276ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:47.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.447+0000 7f54276ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:47.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.447+0000 7f54276ae640 1 -- 172.21.15.19:0/838608901 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5420074cc0 con 0x7f542013f710 2024-03-25T19:39:47.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.534+0000 7f5415ffb640 1 -- 172.21.15.19:0/838608901 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1576595288) 0x7f54100521f0 con 0x7f542013f710 2024-03-25T19:39:47.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.534+0000 7f5415ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:47.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.534+0000 7f5415ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:47.535 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:39:47.535 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:38:58.470042+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:58.470042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:47.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 10 librados: watch_flush enter 2024-03-25T19:39:47.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 10 librados: watch_flush exit 2024-03-25T19:39:47.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T19:39:47.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 10 client.4707.objecter successfully canceled tick 2024-03-25T19:39:47.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 1 -- 172.21.15.19:0/838608901 >> v1:172.21.15.188:6800/2207506229 conn(0x7f540406df70 legacy=0x7f5404070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:47.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 10 monclient: shutdown 2024-03-25T19:39:47.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:47.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 1 -- 172.21.15.19:0/838608901 >> v1:172.21.15.19:6789/0 conn(0x7f542013f710 legacy=0x7f54201d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:47.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 1 -- 172.21.15.19:0/838608901 shutdown_connections 2024-03-25T19:39:47.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 1 -- 172.21.15.19:0/838608901 >> 172.21.15.19:0/838608901 conn(0x7f54200b7ea0 msgr2=0x7f5420074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:47.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 1 -- 172.21.15.19:0/838608901 shutdown_connections 2024-03-25T19:39:47.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 1 -- 172.21.15.19:0/838608901 wait complete. 2024-03-25T19:39:47.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:47.537+0000 7f54276ae640 1 librados: shutdown 2024-03-25T19:39:47.550 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:54.356 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:54.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.469+0000 7fc8d3dcf640 10 monclient: get_monmap_and_config 2024-03-25T19:39:54.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.469+0000 7fc8d3dcf640 10 monclient: build_initial_monmap 2024-03-25T19:39:54.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.469+0000 7fc8d3dcf640 10 monclient: monmap: 2024-03-25T19:39:54.470 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:54.470 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:54.470 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:39:54.470828+0000 2024-03-25T19:39:54.470 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:39:54.470828+0000 2024-03-25T19:39:54.470 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:54.470 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:54.470 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:54.470 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:39:54.470 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:39:54.470 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:54.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.470+0000 7fc8d3dcf640 1 Processor -- start 2024-03-25T19:39:54.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.470+0000 7fc8d3dcf640 1 -- start start 2024-03-25T19:39:54.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.470+0000 7fc8d3dcf640 10 monclient: init 2024-03-25T19:39:54.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.471+0000 7fc8d3dcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:54.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.471+0000 7fc8d3dcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:54.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.471+0000 7fc8d3dcf640 10 monclient(hunting): picked mon.noname-a con 0x7fc8cc0a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:54.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.471+0000 7fc8d3dcf640 10 monclient(hunting): picked mon.noname-b con 0x7fc8cc13f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:54.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.471+0000 7fc8d3dcf640 10 monclient(hunting): picked mon.noname-c con 0x7fc8cc1430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:54.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.471+0000 7fc8d3dcf640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc142520 con 0x7fc8cc0a98f0 2024-03-25T19:39:54.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.471+0000 7fc8d3dcf640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc0b28a0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.471+0000 7fc8d3dcf640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc0b2a30 con 0x7fc8cc13f710 2024-03-25T19:39:54.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.471+0000 7fc8d3dcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:54.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.471+0000 7fc8d3dcf640 10 monclient(hunting): authenticate will time out at 6647.741437s 2024-03-25T19:39:54.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.479+0000 7fc8d236c640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7fc8cc1430c0 0x7fc8cc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:60004/0 (socket says 172.21.15.19:60004) 2024-03-25T19:39:54.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.479+0000 7fc8d236c640 1 -- 172.21.15.19:0/3155324504 learned_addr learned my addr 172.21.15.19:0/3155324504 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:54.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.501+0000 7fc8d0b69640 1 -- 172.21.15.19:0/3155324504 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634815933 0 0) 0x7fc8cc0b2a30 con 0x7fc8cc13f710 2024-03-25T19:39:54.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.501+0000 7fc8d0b69640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.501+0000 7fc8d0b69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.501+0000 7fc8d0b69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.501+0000 7fc8d0b69640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:39:54.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.501+0000 7fc8d0b69640 1 -- 172.21.15.19:0/3155324504 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b8003520 con 0x7fc8cc13f710 2024-03-25T19:39:54.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.502+0000 7fc8d0b69640 1 -- 172.21.15.19:0/3155324504 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602245664 0 0) 0x7fc8cc0b28a0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.502+0000 7fc8d0b69640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.502+0000 7fc8d0b69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.502+0000 7fc8d0b69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.502+0000 7fc8d0b69640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:39:54.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.502+0000 7fc8d0b69640 1 -- 172.21.15.19:0/3155324504 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8cc0b2a30 con 0x7fc8cc1430c0 2024-03-25T19:39:54.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 1 -- 172.21.15.19:0/3155324504 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2368818820 0 0) 0x7fc8b8003520 con 0x7fc8cc13f710 2024-03-25T19:39:54.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 1 -- 172.21.15.19:0/3155324504 >> v1:172.21.15.188:6789/0 conn(0x7fc8cc1430c0 legacy=0x7fc8cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 1 -- 172.21.15.19:0/3155324504 >> v1:172.21.15.19:6789/0 conn(0x7fc8cc0a98f0 legacy=0x7fc8cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 10 monclient: _finish_hunting 0 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 1 monclient: found mon.noname-b 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 1 -- 172.21.15.19:0/3155324504 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc0b44c0 con 0x7fc8cc13f710 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 10 monclient: _finish_auth 0 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 10 monclient: _check_auth_tickets 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 1 -- 172.21.15.19:0/3155324504 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc8bc001690 con 0x7fc8cc13f710 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d3dcf640 5 monclient: authenticate success, global_id 4685 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d3dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d0b69640 10 monclient: dump: 2024-03-25T19:39:54.515 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:54.516 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:54.516 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:54.516 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:54.516 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.516 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:54.516 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:54.516 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:54.516 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:54.516 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:54.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.514+0000 7fc8d3dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:54.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.524+0000 7fc8d0b69640 1 -- 172.21.15.19:0/3155324504 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8bc0017f0 con 0x7fc8cc13f710 2024-03-25T19:39:54.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.524+0000 7fc8d0b69640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:54.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.524+0000 7fc8d0b69640 1 -- 172.21.15.19:0/3155324504 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc8bc001c00 con 0x7fc8cc13f710 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.524+0000 7fc8d3dcf640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.524+0000 7fc8d3dcf640 4 set_mon_vals no callback set 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.524+0000 7fc8d0b69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.524+0000 7fc8d0b69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.524+0000 7fc8d0b69640 10 monclient: dump: 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:54.525 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:54.526 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:54.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.525+0000 7fc8d3dcf640 10 monclient: shutdown 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.525+0000 7fc8d3dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.525+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/3155324504 >> v1:172.21.15.19:6790/0 conn(0x7fc8cc13f710 legacy=0x7fc8cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.525+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/3155324504 shutdown_connections 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.525+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/3155324504 >> 172.21.15.19:0/3155324504 conn(0x7fc8cc0b7ea0 msgr2=0x7fc8cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.525+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/3155324504 shutdown_connections 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.525+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/3155324504 wait complete. 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.526+0000 7fc8d3dcf640 10 monclient: build_initial_monmap 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.526+0000 7fc8d3dcf640 10 monclient: monmap: 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:39:54.528 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.526+0000 7fc8d3dcf640 1 librados: starting msgr at 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.526+0000 7fc8d3dcf640 1 librados: starting objecter 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.526+0000 7fc8d3dcf640 1 Processor -- start 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.527+0000 7fc8d3dcf640 1 -- start start 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.527+0000 7fc8d3dcf640 1 librados: setting wanted keys 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.527+0000 7fc8d3dcf640 1 librados: calling monclient init 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.527+0000 7fc8d3dcf640 10 monclient: init 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.527+0000 7fc8d3dcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:54.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.527+0000 7fc8d3dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:39:54.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.527+0000 7fc8d3dcf640 10 monclient(hunting): picked mon.noname-b con 0x7fc8cc0a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:39:54.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.528+0000 7fc8d3dcf640 10 monclient(hunting): picked mon.noname-c con 0x7fc8cc13f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:39:54.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.528+0000 7fc8d3dcf640 10 monclient(hunting): picked mon.noname-a con 0x7fc8cc1430c0 addr v1:172.21.15.19:6789/0 2024-03-25T19:39:54.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.528+0000 7fc8d3dcf640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc137df0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.528+0000 7fc8d3dcf640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc1337a0 con 0x7fc8cc0a98f0 2024-03-25T19:39:54.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.528+0000 7fc8d3dcf640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8cc133930 con 0x7fc8cc13f710 2024-03-25T19:39:54.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.528+0000 7fc8d3dcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:54.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.528+0000 7fc8d3dcf640 10 monclient(hunting): authenticate will time out at 6647.798050s 2024-03-25T19:39:54.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.533+0000 7fc8d136a640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fc8cc13f710 0x7fc8cc1376e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:33590/0 (socket says 172.21.15.19:33590) 2024-03-25T19:39:54.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.533+0000 7fc8d136a640 1 -- 172.21.15.19:0/4216783363 learned_addr learned my addr 172.21.15.19:0/4216783363 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:39:54.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.534+0000 7fc8c27fc640 10 client.?.objecter ms_handle_connect 0x7fc8cc13f710 2024-03-25T19:39:54.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.534+0000 7fc8c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:54.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.540+0000 7fc8c27fc640 10 client.?.objecter ms_handle_connect 0x7fc8cc1430c0 2024-03-25T19:39:54.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.540+0000 7fc8c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.547+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420369867 0 0) 0x7fc8cc137df0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.547+0000 7fc8c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.547+0000 7fc8c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.547+0000 7fc8c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.547+0000 7fc8c27fc640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:39:54.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.547+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b0004eb0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.551+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878257748 0 0) 0x7fc8cc133930 con 0x7fc8cc13f710 2024-03-25T19:39:54.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.551+0000 7fc8c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.551+0000 7fc8c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.551+0000 7fc8c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.551+0000 7fc8c27fc640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.551+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8cc137df0 con 0x7fc8cc13f710 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.551+0000 7fc8c27fc640 10 client.?.objecter ms_handle_connect 0x7fc8cc0a98f0 2024-03-25T19:39:54.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.551+0000 7fc8c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:54.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.561+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084367257 0 0) 0x7fc8cc1337a0 con 0x7fc8cc0a98f0 2024-03-25T19:39:54.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.561+0000 7fc8c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:54.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.561+0000 7fc8c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:54.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.561+0000 7fc8c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:54.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.561+0000 7fc8c27fc640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:39:54.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.561+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8cc133930 con 0x7fc8cc0a98f0 2024-03-25T19:39:54.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.565+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 986475457 0 0) 0x7fc8b0004eb0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.566 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.565+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8cc1337a0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.566 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.566+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc8c4001bd0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.566 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.566+0000 7fc8c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.566 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.566+0000 7fc8c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:54.566 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.566+0000 7fc8c27fc640 10 monclient(hunting): dump: 2024-03-25T19:39:54.566 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:54.566 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:54.566 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:54.566 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:54.566 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.566 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:54.566 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:54.567 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:54.567 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:54.567 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:54.567 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.566+0000 7fc8c27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:54.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.570+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 196808673 0 0) 0x7fc8cc137df0 con 0x7fc8cc13f710 2024-03-25T19:39:54.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.570+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8b0004eb0 con 0x7fc8cc13f710 2024-03-25T19:39:54.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.570+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc8bc001690 con 0x7fc8cc13f710 2024-03-25T19:39:54.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.570+0000 7fc8c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.570+0000 7fc8c27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:54.570 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.570+0000 7fc8c27fc640 10 monclient(hunting): dump: 2024-03-25T19:39:54.570 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:54.570 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:54.570 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:54.571 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:54.571 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.571 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:54.571 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:54.571 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:54.571 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:54.571 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:54.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.578+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1045173465 0 0) 0x7fc8cc133930 con 0x7fc8cc0a98f0 2024-03-25T19:39:54.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.579+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8cc137df0 con 0x7fc8cc0a98f0 2024-03-25T19:39:54.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.579+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc8c80017e0 con 0x7fc8cc0a98f0 2024-03-25T19:39:54.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.579+0000 7fc8c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.579+0000 7fc8c27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:54.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.579+0000 7fc8c27fc640 10 monclient(hunting): dump: 2024-03-25T19:39:54.579 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:54.579 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:54.579 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:54.579 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:54.579 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.579 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:54.580 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:54.580 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:54.580 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:54.580 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:54.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 474972159 0 0) 0x7fc8cc1337a0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 >> v1:172.21.15.19:6790/0 conn(0x7fc8cc13f710 legacy=0x7fc8cc1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 >> v1:172.21.15.188:6789/0 conn(0x7fc8cc0a98f0 legacy=0x7fc8cc1392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:54.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8c27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:39:54.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8c27fc640 1 monclient: found mon.a 2024-03-25T19:39:54.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:54.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8c27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc1367d0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8c27fc640 10 monclient: _finish_auth 0 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8c27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8d3dcf640 5 monclient: authenticate success, global_id 4713 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8d3dcf640 10 monclient: _renew_subs 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8d3dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/4216783363 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8cc1369e0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8d3dcf640 10 client.4713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8d3dcf640 10 monclient: _renew_subs 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8d3dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/4216783363 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc13b640 con 0x7fc8cc1430c0 2024-03-25T19:39:54.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.595+0000 7fc8d3dcf640 1 librados: init done 2024-03-25T19:39:54.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.596+0000 7fc8d3dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:54.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.596+0000 7fc8d3dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:54.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.596+0000 7fc8d3dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:54.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.596+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/4216783363 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8cc074cc0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.906 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.906+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8c40020c0 con 0x7fc8cc1430c0 2024-03-25T19:39:54.907 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.906+0000 7fc8c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:54.907 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.906+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7fc8c4004d00 con 0x7fc8cc1430c0 2024-03-25T19:39:54.907 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.906+0000 7fc8c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:54.907 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.906+0000 7fc8c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:54.907 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.906+0000 7fc8c27fc640 10 monclient: dump: 2024-03-25T19:39:54.907 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:39:54.907 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:39:54.907 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:54.907 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:39:54.907 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:54.907 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.906+0000 7fc8c3fff640 4 set_mon_vals no callback set 2024-03-25T19:39:54.908 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.907+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7fc8c401ad50 con 0x7fc8cc1430c0 2024-03-25T19:39:54.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.908+0000 7fc8d1b6b640 10 monclient: get_auth_request con 0x7fc8b006e0b0 auth_method 0 2024-03-25T19:39:54.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.908+0000 7fc8c27fc640 10 client.4713.objecter ms_handle_connect 0x7fc8b006e0b0 2024-03-25T19:39:54.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.913+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1490778728 0 0) 0x7fc8c4054a40 con 0x7fc8cc1430c0 2024-03-25T19:39:54.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.913+0000 7fc8c27fc640 10 client.4713.objecter ms_dispatch 0x7fc8cc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:39:54.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.913+0000 7fc8c27fc640 3 client.4713.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:39:54.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.913+0000 7fc8c27fc640 3 client.4713.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:39:54.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.913+0000 7fc8c27fc640 20 client.4713.objecter dump_active .. 0 homeless 2024-03-25T19:39:54.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.913+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8c4050380 con 0x7fc8cc1430c0 2024-03-25T19:39:54.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.913+0000 7fc8c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:54.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:54.913+0000 7fc8c27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:55.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.139+0000 7fc8d3dcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:55.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.139+0000 7fc8d3dcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:55.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.139+0000 7fc8d3dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:39:55.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.139+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/4216783363 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8cc0627b0 con 0x7fc8cc1430c0 2024-03-25T19:39:55.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.169+0000 7fc8c27fc640 1 -- 172.21.15.19:0/4216783363 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1576595288) 0x7fc8c4052480 con 0x7fc8cc1430c0 2024-03-25T19:39:55.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.169+0000 7fc8c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:55.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.169+0000 7fc8c27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:55.169 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:39:55.170 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:38:58.470042+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:58.470042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:55.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 10 librados: watch_flush enter 2024-03-25T19:39:55.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 10 librados: watch_flush exit 2024-03-25T19:39:55.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 20 client.4713.objecter shutdown clearing up homeless session... 2024-03-25T19:39:55.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 10 client.4713.objecter successfully canceled tick 2024-03-25T19:39:55.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/4216783363 >> v1:172.21.15.188:6800/2207506229 conn(0x7fc8b006e0b0 legacy=0x7fc8b0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:55.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 10 monclient: shutdown 2024-03-25T19:39:55.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:55.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/4216783363 >> v1:172.21.15.19:6789/0 conn(0x7fc8cc1430c0 legacy=0x7fc8cc133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:55.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/4216783363 shutdown_connections 2024-03-25T19:39:55.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/4216783363 >> 172.21.15.19:0/4216783363 conn(0x7fc8cc0b7ea0 msgr2=0x7fc8cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:55.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/4216783363 shutdown_connections 2024-03-25T19:39:55.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 1 -- 172.21.15.19:0/4216783363 wait complete. 2024-03-25T19:39:55.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:39:55.172+0000 7fc8d3dcf640 1 librados: shutdown 2024-03-25T19:39:55.183 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:58.764 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:21.494 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.771 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:21.501 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.809 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:21.538 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.837 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:11:21.540 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.843 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:11:21.547 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.856 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:21.586 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.864 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:11:21.567 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.864 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:21.594 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:58.886 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:21.616 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:01.354 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:24.083 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:01.357 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:24.087 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:01.379 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:11:24.082 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:01.380 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:24.109 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:01.382 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:11:24.085 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:01.383 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:24.112 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:01.384 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:24.114 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:01.399 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:11:24.102 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:01.399 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:24.129 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:01.990 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:02.104 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.104+0000 7f23cf6f8640 10 monclient: get_monmap_and_config 2024-03-25T19:40:02.104 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.104+0000 7f23cf6f8640 10 monclient: build_initial_monmap 2024-03-25T19:40:02.104 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.104+0000 7f23cf6f8640 10 monclient: monmap: 2024-03-25T19:40:02.104 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:40:02.104 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:02.104 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:40:02.105475+0000 2024-03-25T19:40:02.104 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:40:02.105475+0000 2024-03-25T19:40:02.104 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:02.104 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:02.105 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:40:02.105 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:40:02.105 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:40:02.105 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:02.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.105+0000 7f23cf6f8640 1 Processor -- start 2024-03-25T19:40:02.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.105+0000 7f23cf6f8640 1 -- start start 2024-03-25T19:40:02.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.105+0000 7f23cf6f8640 10 monclient: init 2024-03-25T19:40:02.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.106+0000 7f23cf6f8640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:02.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.106+0000 7f23cf6f8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:02.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.106+0000 7f23cf6f8640 10 monclient(hunting): picked mon.noname-a con 0x7f23c80a98f0 addr v1:172.21.15.19:6789/0 2024-03-25T19:40:02.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.106+0000 7f23cf6f8640 10 monclient(hunting): picked mon.noname-b con 0x7f23c813f710 addr v1:172.21.15.19:6790/0 2024-03-25T19:40:02.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.106+0000 7f23cf6f8640 10 monclient(hunting): picked mon.noname-c con 0x7f23c81430c0 addr v1:172.21.15.188:6789/0 2024-03-25T19:40:02.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.106+0000 7f23cf6f8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c8142520 con 0x7f23c80a98f0 2024-03-25T19:40:02.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.106+0000 7f23cf6f8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80b28a0 con 0x7f23c81430c0 2024-03-25T19:40:02.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.106+0000 7f23cf6f8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80b2a30 con 0x7f23c813f710 2024-03-25T19:40:02.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.106+0000 7f23cf6f8640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:02.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.106+0000 7f23cf6f8640 10 monclient(hunting): authenticate will time out at 6655.376415s 2024-03-25T19:40:02.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.108+0000 7f23ccc93640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f23c813f710 0x7f23c81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:43280/0 (socket says 172.21.15.19:43280) 2024-03-25T19:40:02.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.108+0000 7f23ccc93640 1 -- 172.21.15.19:0/83000096 learned_addr learned my addr 172.21.15.19:0/83000096 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:40:02.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.183+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72343680 0 0) 0x7f23c80b2a30 con 0x7f23c813f710 2024-03-25T19:40:02.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.183+0000 7f23bffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:02.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.183+0000 7f23bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:02.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.183+0000 7f23bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:02.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.183+0000 7f23bffff640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:40:02.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.183+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b4003520 con 0x7f23c813f710 2024-03-25T19:40:02.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.185+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845892512 0 0) 0x7f23c80b28a0 con 0x7f23c81430c0 2024-03-25T19:40:02.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.185+0000 7f23bffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:02.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.185+0000 7f23bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:02.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.185+0000 7f23bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:02.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.185+0000 7f23bffff640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:40:02.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.185+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c80b2a30 con 0x7f23c81430c0 2024-03-25T19:40:02.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.190+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123604090 0 0) 0x7f23c8142520 con 0x7f23c80a98f0 2024-03-25T19:40:02.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.190+0000 7f23bffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:02.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.190+0000 7f23bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:02.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.190+0000 7f23bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:02.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.190+0000 7f23bffff640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:40:02.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.190+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c80b28a0 con 0x7f23c80a98f0 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1041867797 0 0) 0x7f23c80b2a30 con 0x7f23c81430c0 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 >> v1:172.21.15.19:6790/0 conn(0x7f23c813f710 legacy=0x7f23c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 >> v1:172.21.15.19:6789/0 conn(0x7f23c80a98f0 legacy=0x7f23c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23bffff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23bffff640 1 monclient: found mon.noname-c 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23bffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c80b44c0 con 0x7f23c81430c0 2024-03-25T19:40:02.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23bffff640 10 monclient: _finish_auth 0 2024-03-25T19:40:02.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23bffff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:02.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:02.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23cf6f8640 5 monclient: authenticate success, global_id 4714 2024-03-25T19:40:02.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.197+0000 7f23cf6f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.200+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f23c00019b0 con 0x7f23c81430c0 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.200+0000 7f23bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.200+0000 7f23bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.200+0000 7f23bffff640 10 monclient: dump: 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:02.201 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:02.202 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.201+0000 7f23cf6f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:02.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.212+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23c0001c00 con 0x7f23c81430c0 2024-03-25T19:40:02.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.212+0000 7f23bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:02.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.212+0000 7f23bffff640 1 -- 172.21.15.19:0/83000096 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f23c0001f90 con 0x7f23c81430c0 2024-03-25T19:40:02.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.212+0000 7f23bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:02.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.212+0000 7f23bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:02.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.212+0000 7f23bffff640 10 monclient: dump: 2024-03-25T19:40:02.213 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:02.213 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:02.213 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:02.213 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:02.213 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:02.213 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:02.213 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:02.213 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:02.213 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:02.213 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:02.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.212+0000 7f23cf6f8640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:02.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.212+0000 7f23cf6f8640 4 set_mon_vals no callback set 2024-03-25T19:40:02.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.213+0000 7f23cf6f8640 10 monclient: shutdown 2024-03-25T19:40:02.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.213+0000 7f23cf6f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:02.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.213+0000 7f23cf6f8640 1 -- 172.21.15.19:0/83000096 >> v1:172.21.15.188:6789/0 conn(0x7f23c81430c0 legacy=0x7f23c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.213+0000 7f23cf6f8640 1 -- 172.21.15.19:0/83000096 shutdown_connections 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.213+0000 7f23cf6f8640 1 -- 172.21.15.19:0/83000096 >> 172.21.15.19:0/83000096 conn(0x7f23c80b7ea0 msgr2=0x7f23c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.214+0000 7f23cf6f8640 1 -- 172.21.15.19:0/83000096 shutdown_connections 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.214+0000 7f23cf6f8640 1 -- 172.21.15.19:0/83000096 wait complete. 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.215+0000 7f23cf6f8640 10 monclient: build_initial_monmap 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.215+0000 7f23cf6f8640 10 monclient: monmap: 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:40:02.216 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:40:02.217 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:40:02.217 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:40:02.217 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:02.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.215+0000 7f23cf6f8640 1 librados: starting msgr at 2024-03-25T19:40:02.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.215+0000 7f23cf6f8640 1 librados: starting objecter 2024-03-25T19:40:02.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.215+0000 7f23cf6f8640 1 Processor -- start 2024-03-25T19:40:02.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.215+0000 7f23cf6f8640 1 -- start start 2024-03-25T19:40:02.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.215+0000 7f23cf6f8640 1 librados: setting wanted keys 2024-03-25T19:40:02.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.215+0000 7f23cf6f8640 1 librados: calling monclient init 2024-03-25T19:40:02.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.215+0000 7f23cf6f8640 10 monclient: init 2024-03-25T19:40:02.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.216+0000 7f23cf6f8640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:02.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.216+0000 7f23cf6f8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.216+0000 7f23cf6f8640 10 monclient(hunting): picked mon.noname-b con 0x7f23c80a98f0 addr v1:172.21.15.188:6789/0 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.216+0000 7f23cf6f8640 10 monclient(hunting): picked mon.noname-a con 0x7f23c813f710 addr v1:172.21.15.19:6789/0 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.216+0000 7f23cf6f8640 10 monclient(hunting): picked mon.noname-c con 0x7f23c81430c0 addr v1:172.21.15.19:6790/0 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.216+0000 7f23cf6f8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c8138f00 con 0x7f23c813f710 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.216+0000 7f23cf6f8640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c81377e0 con 0x7f23c80a98f0 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.216+0000 7f23cf6f8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c81dc070 con 0x7f23c81430c0 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.216+0000 7f23cf6f8640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:02.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.216+0000 7f23cf6f8640 10 monclient(hunting): authenticate will time out at 6655.486702s 2024-03-25T19:40:02.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.221+0000 7f23cd494640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7f23c80a98f0 0x7f23c8137c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:40374/0 (socket says 172.21.15.19:40374) 2024-03-25T19:40:02.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.221+0000 7f23cd494640 1 -- 172.21.15.19:0/3799087648 learned_addr learned my addr 172.21.15.19:0/3799087648 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:40:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.222+0000 7f23bdffb640 10 client.?.objecter ms_handle_connect 0x7f23c80a98f0 2024-03-25T19:40:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.222+0000 7f23bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:02.230 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.230+0000 7f23bdffb640 10 client.?.objecter ms_handle_connect 0x7f23c81430c0 2024-03-25T19:40:02.230 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.230+0000 7f23bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:02.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.238+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035608976 0 0) 0x7f23c81dc070 con 0x7f23c81430c0 2024-03-25T19:40:02.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.238+0000 7f23bdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:02.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.238+0000 7f23bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:02.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.238+0000 7f23bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:02.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.238+0000 7f23bdffb640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:40:02.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.238+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ac004eb0 con 0x7f23c81430c0 2024-03-25T19:40:02.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.244+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163414641 0 0) 0x7f23c81377e0 con 0x7f23c80a98f0 2024-03-25T19:40:02.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.244+0000 7f23bdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:02.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.244+0000 7f23bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:02.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.244+0000 7f23bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:02.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.244+0000 7f23bdffb640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:40:02.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.244+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c81dc070 con 0x7f23c80a98f0 2024-03-25T19:40:02.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.247+0000 7f23bdffb640 10 client.?.objecter ms_handle_connect 0x7f23c813f710 2024-03-25T19:40:02.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.247+0000 7f23bdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:02.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.257+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3338294868 0 0) 0x7f23ac004eb0 con 0x7f23c81430c0 2024-03-25T19:40:02.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.257+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23c81377e0 con 0x7f23c81430c0 2024-03-25T19:40:02.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.257+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f23c0001f10 con 0x7f23c81430c0 2024-03-25T19:40:02.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.257+0000 7f23bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:02.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.257+0000 7f23bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:02.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.257+0000 7f23bdffb640 10 monclient(hunting): dump: 2024-03-25T19:40:02.257 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:02.257 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:02.258 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:02.258 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:02.258 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:02.258 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:02.258 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:02.258 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:02.258 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:02.258 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:02.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.257+0000 7f23bdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:02.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.258+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1414362306 0 0) 0x7f23c81dc070 con 0x7f23c80a98f0 2024-03-25T19:40:02.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.258+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 --> v1:172.21.15.188:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23ac004eb0 con 0x7f23c80a98f0 2024-03-25T19:40:02.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.258+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f23c40019f0 con 0x7f23c80a98f0 2024-03-25T19:40:02.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.258+0000 7f23bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:02.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.258+0000 7f23bdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:02.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.258+0000 7f23bdffb640 10 monclient(hunting): dump: 2024-03-25T19:40:02.259 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:02.259 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:02.260 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:02.260 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:02.260 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:02.260 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:02.260 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:02.260 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:02.260 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:02.260 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:02.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.262+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695831184 0 0) 0x7f23c8138f00 con 0x7f23c813f710 2024-03-25T19:40:02.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.262+0000 7f23bdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:02.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.262+0000 7f23bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:02.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.262+0000 7f23bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:02.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.262+0000 7f23bdffb640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:40:02.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.262+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c81dc070 con 0x7f23c813f710 2024-03-25T19:40:02.279 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.1 v1:172.21.15.188:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3604358928 0 0) 0x7f23ac004eb0 con 0x7f23c80a98f0 2024-03-25T19:40:02.279 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 >> v1:172.21.15.19:6790/0 conn(0x7f23c81430c0 legacy=0x7f23c81db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 >> v1:172.21.15.19:6789/0 conn(0x7f23c813f710 legacy=0x7f23c81db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23bdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23bdffb640 1 monclient: found mon.b 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23bdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c8139150 con 0x7f23c80a98f0 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23bdffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23bdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23cf6f8640 5 monclient: authenticate success, global_id 4717 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23cf6f8640 10 monclient: _renew_subs 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23cf6f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:40:02.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23cf6f8640 1 -- 172.21.15.19:0/3799087648 --> v1:172.21.15.188:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c8139430 con 0x7f23c80a98f0 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23cf6f8640 10 client.4717.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23cf6f8640 10 monclient: _renew_subs 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23cf6f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23cf6f8640 1 -- 172.21.15.19:0/3799087648 --> v1:172.21.15.188:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c8136dd0 con 0x7f23c80a98f0 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.279+0000 7f23cf6f8640 1 librados: init done 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.280+0000 7f23cf6f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.280+0000 7f23cf6f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:02.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.280+0000 7f23cf6f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:40:02.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.280+0000 7f23cf6f8640 1 -- 172.21.15.19:0/3799087648 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c80b3ef0 con 0x7f23c80a98f0 2024-03-25T19:40:02.554 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.554+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.1 v1:172.21.15.188:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23c4003610 con 0x7f23c80a98f0 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.554+0000 7f23bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.554+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.1 v1:172.21.15.188:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f23c40039e0 con 0x7f23c80a98f0 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.554+0000 7f23bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.554+0000 7f23bdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.554+0000 7f23bdffb640 10 monclient: dump: 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:02.555 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:02.556 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:02.556 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:02.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.554+0000 7f23bf7fe640 4 set_mon_vals no callback set 2024-03-25T19:40:02.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.555+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.1 v1:172.21.15.188:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f23c4003c00 con 0x7f23c80a98f0 2024-03-25T19:40:02.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.556+0000 7f23ccc93640 10 monclient: get_auth_request con 0x7f23ac06dd20 auth_method 0 2024-03-25T19:40:02.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.556+0000 7f23bdffb640 10 client.4717.objecter ms_handle_connect 0x7f23ac06dd20 2024-03-25T19:40:02.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.560+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.1 v1:172.21.15.188:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1490778728 0 0) 0x7f23c4054810 con 0x7f23c80a98f0 2024-03-25T19:40:02.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.561+0000 7f23bdffb640 10 client.4717.objecter ms_dispatch 0x7f23c8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:40:02.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.561+0000 7f23bdffb640 3 client.4717.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:40:02.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.561+0000 7f23bdffb640 3 client.4717.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:40:02.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.561+0000 7f23bdffb640 20 client.4717.objecter dump_active .. 0 homeless 2024-03-25T19:40:02.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.564+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.1 v1:172.21.15.188:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23c4050320 con 0x7f23c80a98f0 2024-03-25T19:40:02.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.564+0000 7f23bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:02.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.564+0000 7f23bdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:02.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.837+0000 7f23cf6f8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:02.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.837+0000 7f23cf6f8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:02.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.837+0000 7f23cf6f8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.188:6789/0 2024-03-25T19:40:02.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.837+0000 7f23cf6f8640 1 -- 172.21.15.19:0/3799087648 --> v1:172.21.15.188:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23c81375b0 con 0x7f23c80a98f0 2024-03-25T19:40:02.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.863+0000 7f23bdffb640 1 -- 172.21.15.19:0/3799087648 <== mon.1 v1:172.21.15.188:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1576595288) 0x7f23c81375b0 con 0x7f23c80a98f0 2024-03-25T19:40:02.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.863+0000 7f23bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:02.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.863+0000 7f23bdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:02.863 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:40:02.863 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:38:58.470042+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:58.470042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:02.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 10 librados: watch_flush enter 2024-03-25T19:40:02.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 10 librados: watch_flush exit 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 20 client.4717.objecter shutdown clearing up homeless session... 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 10 client.4717.objecter successfully canceled tick 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 1 -- 172.21.15.19:0/3799087648 >> v1:172.21.15.188:6800/2207506229 conn(0x7f23ac06dd20 legacy=0x7f23ac0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 10 monclient: shutdown 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 1 -- 172.21.15.19:0/3799087648 >> v1:172.21.15.188:6789/0 conn(0x7f23c80a98f0 legacy=0x7f23c8137c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 1 -- 172.21.15.19:0/3799087648 shutdown_connections 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 1 -- 172.21.15.19:0/3799087648 >> 172.21.15.19:0/3799087648 conn(0x7f23c80b7ea0 msgr2=0x7f23c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 1 -- 172.21.15.19:0/3799087648 shutdown_connections 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 1 -- 172.21.15.19:0/3799087648 wait complete. 2024-03-25T19:40:02.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:02.865+0000 7f23cf6f8640 1 librados: shutdown 2024-03-25T19:40:02.878 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:09.685 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:09.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.802+0000 7ff4b5154640 10 monclient: get_monmap_and_config 2024-03-25T19:40:09.802 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.802+0000 7ff4b5154640 10 monclient: build_initial_monmap 2024-03-25T19:40:09.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.802+0000 7ff4b5154640 10 monclient: monmap: 2024-03-25T19:40:09.803 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:40:09.803 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:09.803 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:40:09.804025+0000 2024-03-25T19:40:09.803 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:40:09.804025+0000 2024-03-25T19:40:09.803 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:09.803 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:09.803 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:40:09.803 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:40:09.803 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:40:09.803 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:09.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.803+0000 7ff4b5154640 1 Processor -- start 2024-03-25T19:40:09.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.803+0000 7ff4b5154640 1 -- start start 2024-03-25T19:40:09.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.803+0000 7ff4b5154640 10 monclient: init 2024-03-25T19:40:09.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.804+0000 7ff4b5154640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:09.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.804+0000 7ff4b5154640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:09.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.804+0000 7ff4b5154640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b00b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:40:09.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.804+0000 7ff4b5154640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b00b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:40:09.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.804+0000 7ff4b5154640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b014be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:40:09.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.804+0000 7ff4b5154640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b00af940 con 0x7ff4b00b1730 2024-03-25T19:40:09.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.804+0000 7ff4b5154640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b00b0a70 con 0x7ff4b00b4d60 2024-03-25T19:40:09.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.804+0000 7ff4b5154640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b00b0d50 con 0x7ff4b014be90 2024-03-25T19:40:09.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.804+0000 7ff4b5154640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:09.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.804+0000 7ff4b5154640 10 monclient(hunting): authenticate will time out at 6663.074690s 2024-03-25T19:40:09.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.807+0000 7ff4aed9d640 1 --1- >> v1:172.21.15.188:6789/0 conn(0x7ff4b00b4d60 0x7ff4b013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.188:6789/0 says I am v1:172.21.15.19:32990/0 (socket says 172.21.15.19:32990) 2024-03-25T19:40:09.806 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.807+0000 7ff4aed9d640 1 -- 172.21.15.19:0/4103787310 learned_addr learned my addr 172.21.15.19:0/4103787310 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:40:09.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.819+0000 7ff4add9b640 1 -- 172.21.15.19:0/4103787310 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704397660 0 0) 0x7ff4b00b0a70 con 0x7ff4b00b4d60 2024-03-25T19:40:09.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.819+0000 7ff4add9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.819+0000 7ff4add9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.819+0000 7ff4add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.819+0000 7ff4add9b640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:40:09.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.819+0000 7ff4add9b640 1 -- 172.21.15.19:0/4103787310 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff498003520 con 0x7ff4b00b4d60 2024-03-25T19:40:09.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.826+0000 7ff4add9b640 1 -- 172.21.15.19:0/4103787310 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302573985 0 0) 0x7ff4b00af940 con 0x7ff4b00b1730 2024-03-25T19:40:09.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.826+0000 7ff4add9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.826+0000 7ff4add9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.826+0000 7ff4add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.826+0000 7ff4add9b640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:40:09.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.826+0000 7ff4add9b640 1 -- 172.21.15.19:0/4103787310 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b00b0a70 con 0x7ff4b00b1730 2024-03-25T19:40:09.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 1 -- 172.21.15.19:0/4103787310 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1511487265 0 0) 0x7ff498003520 con 0x7ff4b00b4d60 2024-03-25T19:40:09.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 1 -- 172.21.15.19:0/4103787310 >> v1:172.21.15.19:6790/0 conn(0x7ff4b014be90 legacy=0x7ff4b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 1 -- 172.21.15.19:0/4103787310 >> v1:172.21.15.19:6789/0 conn(0x7ff4b00b1730 legacy=0x7ff4b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 10 monclient: _finish_hunting 0 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 1 monclient: found mon.noname-c 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 1 -- 172.21.15.19:0/4103787310 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b00b1030 con 0x7ff4b00b4d60 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 10 monclient: _finish_auth 0 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 10 monclient: _check_auth_tickets 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 1 -- 172.21.15.19:0/4103787310 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff4a0001690 con 0x7ff4b00b4d60 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:09.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4add9b640 10 monclient: dump: 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4b5154640 5 monclient: authenticate success, global_id 4720 2024-03-25T19:40:09.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.829+0000 7ff4b5154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:09.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.838+0000 7ff4add9b640 1 -- 172.21.15.19:0/4103787310 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4a00017f0 con 0x7ff4b00b4d60 2024-03-25T19:40:09.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.838+0000 7ff4add9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.838+0000 7ff4add9b640 1 -- 172.21.15.19:0/4103787310 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff4a0001c00 con 0x7ff4b00b4d60 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.838+0000 7ff4add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.838+0000 7ff4add9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.838+0000 7ff4add9b640 10 monclient: dump: 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:09.839 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:09.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.838+0000 7ff4b5154640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:09.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.838+0000 7ff4b5154640 4 set_mon_vals no callback set 2024-03-25T19:40:09.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.839+0000 7ff4b5154640 10 monclient: shutdown 2024-03-25T19:40:09.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.839+0000 7ff4b5154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:09.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.839+0000 7ff4b5154640 1 -- 172.21.15.19:0/4103787310 >> v1:172.21.15.188:6789/0 conn(0x7ff4b00b4d60 legacy=0x7ff4b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.839+0000 7ff4b5154640 1 -- 172.21.15.19:0/4103787310 shutdown_connections 2024-03-25T19:40:09.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.839+0000 7ff4b5154640 1 -- 172.21.15.19:0/4103787310 >> 172.21.15.19:0/4103787310 conn(0x7ff4b00617a0 msgr2=0x7ff4b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:09.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.839+0000 7ff4b5154640 1 -- 172.21.15.19:0/4103787310 shutdown_connections 2024-03-25T19:40:09.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.839+0000 7ff4b5154640 1 -- 172.21.15.19:0/4103787310 wait complete. 2024-03-25T19:40:09.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.840+0000 7ff4b5154640 10 monclient: build_initial_monmap 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.840+0000 7ff4b5154640 10 monclient: monmap: 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.841+0000 7ff4b5154640 1 librados: starting msgr at 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.841+0000 7ff4b5154640 1 librados: starting objecter 2024-03-25T19:40:09.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.841+0000 7ff4b5154640 1 Processor -- start 2024-03-25T19:40:09.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.841+0000 7ff4b5154640 1 -- start start 2024-03-25T19:40:09.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.841+0000 7ff4b5154640 1 librados: setting wanted keys 2024-03-25T19:40:09.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.841+0000 7ff4b5154640 1 librados: calling monclient init 2024-03-25T19:40:09.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.841+0000 7ff4b5154640 10 monclient: init 2024-03-25T19:40:09.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.842+0000 7ff4b5154640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:09.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.842+0000 7ff4b5154640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:09.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.842+0000 7ff4b5154640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b00b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:40:09.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.842+0000 7ff4b5154640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b00b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:40:09.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.842+0000 7ff4b5154640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b014be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:40:09.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.842+0000 7ff4b5154640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b013a8d0 con 0x7ff4b00b4d60 2024-03-25T19:40:09.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.842+0000 7ff4b5154640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b01d7840 con 0x7ff4b014be90 2024-03-25T19:40:09.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.842+0000 7ff4b5154640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b01d8470 con 0x7ff4b00b1730 2024-03-25T19:40:09.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.842+0000 7ff4b5154640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:09.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.842+0000 7ff4b5154640 10 monclient(hunting): authenticate will time out at 6663.112269s 2024-03-25T19:40:09.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.843+0000 7ff4ae59c640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7ff4b00b4d60 0x7ff4b01d7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:58454/0 (socket says 172.21.15.19:58454) 2024-03-25T19:40:09.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.843+0000 7ff4ae59c640 1 -- 172.21.15.19:0/2702305109 learned_addr learned my addr 172.21.15.19:0/2702305109 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:40:09.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.846+0000 7ff4977fe640 10 client.?.objecter ms_handle_connect 0x7ff4b00b4d60 2024-03-25T19:40:09.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.846+0000 7ff4977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:09.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.856+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975287938 0 0) 0x7ff4b013a8d0 con 0x7ff4b00b4d60 2024-03-25T19:40:09.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.856+0000 7ff4977fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.856+0000 7ff4977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.856+0000 7ff4977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.856+0000 7ff4977fe640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:40:09.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.856+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff49c004eb0 con 0x7ff4b00b4d60 2024-03-25T19:40:09.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.864+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1599849578 0 0) 0x7ff49c004eb0 con 0x7ff4b00b4d60 2024-03-25T19:40:09.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.864+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4b013a8d0 con 0x7ff4b00b4d60 2024-03-25T19:40:09.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.864+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff4a4001a10 con 0x7ff4b00b4d60 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.864+0000 7ff4977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.864+0000 7ff4977fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.864+0000 7ff4977fe640 10 monclient(hunting): dump: 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.864+0000 7ff4977fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.865+0000 7ff4977fe640 10 client.?.objecter ms_handle_connect 0x7ff4b014be90 2024-03-25T19:40:09.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.865+0000 7ff4977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:09.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.870+0000 7ff4977fe640 10 client.?.objecter ms_handle_connect 0x7ff4b00b1730 2024-03-25T19:40:09.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.870+0000 7ff4977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:09.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.879+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783112953 0 0) 0x7ff4b01d7840 con 0x7ff4b014be90 2024-03-25T19:40:09.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.879+0000 7ff4977fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:09.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.879+0000 7ff4977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:09.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.879+0000 7ff4977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:09.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.879+0000 7ff4977fe640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:40:09.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.879+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff49c004eb0 con 0x7ff4b014be90 2024-03-25T19:40:09.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4070746057 0 0) 0x7ff4b013a8d0 con 0x7ff4b00b4d60 2024-03-25T19:40:09.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 >> v1:172.21.15.19:6790/0 conn(0x7ff4b00b1730 legacy=0x7ff4b0139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 >> v1:172.21.15.188:6789/0 conn(0x7ff4b014be90 legacy=0x7ff4b01e21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:09.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4977fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:09.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4977fe640 1 monclient: found mon.a 2024-03-25T19:40:09.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:09.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4977fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:40:09.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b01d8600 con 0x7ff4b00b4d60 2024-03-25T19:40:09.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4977fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:09.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4977fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:09.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:09.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4b5154640 5 monclient: authenticate success, global_id 4725 2024-03-25T19:40:09.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4b5154640 10 monclient: _renew_subs 2024-03-25T19:40:09.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4b5154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:40:09.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4b5154640 1 -- 172.21.15.19:0/2702305109 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b013afd0 con 0x7ff4b00b4d60 2024-03-25T19:40:09.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4b5154640 10 client.4725.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:09.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4b5154640 10 monclient: _renew_subs 2024-03-25T19:40:09.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4b5154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:40:09.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4b5154640 1 -- 172.21.15.19:0/2702305109 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b0143340 con 0x7ff4b00b4d60 2024-03-25T19:40:09.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.881+0000 7ff4b5154640 1 librados: init done 2024-03-25T19:40:09.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.882+0000 7ff4b5154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:09.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.882+0000 7ff4b5154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:09.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.882+0000 7ff4b5154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:40:09.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:09.882+0000 7ff4b5154640 1 -- 172.21.15.19:0/2702305109 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b00627b0 con 0x7ff4b00b4d60 2024-03-25T19:40:10.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.198+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4a4001d10 con 0x7ff4b00b4d60 2024-03-25T19:40:10.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.198+0000 7ff4977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.198+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7ff4a4003a60 con 0x7ff4b00b4d60 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.198+0000 7ff4977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.198+0000 7ff4977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.198+0000 7ff4977fe640 10 monclient: dump: 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:10.199 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:10.200 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:10.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.198+0000 7ff4ad59a640 4 set_mon_vals no callback set 2024-03-25T19:40:10.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.199+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7ff4a4003c80 con 0x7ff4b00b4d60 2024-03-25T19:40:10.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.199+0000 7ff4aed9d640 10 monclient: get_auth_request con 0x7ff49c06dd40 auth_method 0 2024-03-25T19:40:10.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.199+0000 7ff4977fe640 10 client.4725.objecter ms_handle_connect 0x7ff49c06dd40 2024-03-25T19:40:10.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.207+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1490778728 0 0) 0x7ff4a4054970 con 0x7ff4b00b4d60 2024-03-25T19:40:10.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.207+0000 7ff4977fe640 10 client.4725.objecter ms_dispatch 0x7ff4b0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:40:10.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.207+0000 7ff4977fe640 3 client.4725.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:40:10.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.207+0000 7ff4977fe640 3 client.4725.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:40:10.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.207+0000 7ff4977fe640 20 client.4725.objecter dump_active .. 0 homeless 2024-03-25T19:40:10.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.207+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4a4050480 con 0x7ff4b00b4d60 2024-03-25T19:40:10.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.207+0000 7ff4977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:10.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.207+0000 7ff4977fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:10.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.426+0000 7ff4b5154640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:10.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.426+0000 7ff4b5154640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:10.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.426+0000 7ff4b5154640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:40:10.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.426+0000 7ff4b5154640 1 -- 172.21.15.19:0/2702305109 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4b0069f40 con 0x7ff4b00b4d60 2024-03-25T19:40:10.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.457+0000 7ff4977fe640 1 -- 172.21.15.19:0/2702305109 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1576595288) 0x7ff4a40523b0 con 0x7ff4b00b4d60 2024-03-25T19:40:10.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.457+0000 7ff4977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:10.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.457+0000 7ff4977fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:10.458 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:40:10.458 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:38:58.470042+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:58.470042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:10.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.459+0000 7ff4b5154640 10 librados: watch_flush enter 2024-03-25T19:40:10.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.459+0000 7ff4b5154640 10 librados: watch_flush exit 2024-03-25T19:40:10.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.459+0000 7ff4b5154640 20 client.4725.objecter shutdown clearing up homeless session... 2024-03-25T19:40:10.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.459+0000 7ff4b5154640 10 client.4725.objecter successfully canceled tick 2024-03-25T19:40:10.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.460+0000 7ff4b5154640 1 -- 172.21.15.19:0/2702305109 >> v1:172.21.15.188:6800/2207506229 conn(0x7ff49c06dd40 legacy=0x7ff49c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:10.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.460+0000 7ff4b5154640 10 monclient: shutdown 2024-03-25T19:40:10.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.460+0000 7ff4b5154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:10.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.460+0000 7ff4b5154640 1 -- 172.21.15.19:0/2702305109 >> v1:172.21.15.19:6789/0 conn(0x7ff4b00b4d60 legacy=0x7ff4b01d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:10.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.460+0000 7ff4b5154640 1 -- 172.21.15.19:0/2702305109 shutdown_connections 2024-03-25T19:40:10.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.460+0000 7ff4b5154640 1 -- 172.21.15.19:0/2702305109 >> 172.21.15.19:0/2702305109 conn(0x7ff4b00617a0 msgr2=0x7ff4b00b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:10.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.460+0000 7ff4b5154640 1 -- 172.21.15.19:0/2702305109 shutdown_connections 2024-03-25T19:40:10.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.460+0000 7ff4b5154640 1 -- 172.21.15.19:0/2702305109 wait complete. 2024-03-25T19:40:10.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:10.460+0000 7ff4b5154640 1 librados: shutdown 2024-03-25T19:40:10.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:17.279 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.395+0000 7f65559cf640 10 monclient: get_monmap_and_config 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.395+0000 7f65559cf640 10 monclient: build_initial_monmap 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.395+0000 7f65559cf640 10 monclient: monmap: 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:40:17.396327+0000 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:40:17.396327+0000 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:40:17.395 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:17.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.395+0000 7f65559cf640 1 Processor -- start 2024-03-25T19:40:17.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.395+0000 7f65559cf640 1 -- start start 2024-03-25T19:40:17.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.395+0000 7f65559cf640 10 monclient: init 2024-03-25T19:40:17.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.396+0000 7f65559cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:17.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.396+0000 7f65559cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:17.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.396+0000 7f65559cf640 10 monclient(hunting): picked mon.noname-c con 0x7f65500b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:40:17.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.396+0000 7f65559cf640 10 monclient(hunting): picked mon.noname-a con 0x7f65500b1730 addr v1:172.21.15.19:6789/0 2024-03-25T19:40:17.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.396+0000 7f65559cf640 10 monclient(hunting): picked mon.noname-b con 0x7f655014be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:40:17.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.396+0000 7f65559cf640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65500af940 con 0x7f65500b1730 2024-03-25T19:40:17.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.396+0000 7f65559cf640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65500b0a70 con 0x7f65500b4d60 2024-03-25T19:40:17.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.396+0000 7f65559cf640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65500b0d50 con 0x7f655014be90 2024-03-25T19:40:17.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.396+0000 7f65559cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:17.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.396+0000 7f65559cf640 10 monclient(hunting): authenticate will time out at 6670.666571s 2024-03-25T19:40:17.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.403+0000 7f654e7fc640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f65500b1730 0x7f65500b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:34554/0 (socket says 172.21.15.19:34554) 2024-03-25T19:40:17.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.403+0000 7f654e7fc640 1 -- 172.21.15.19:0/3417303966 learned_addr learned my addr 172.21.15.19:0/3417303966 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:40:17.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.415+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327082558 0 0) 0x7f65500b0a70 con 0x7f65500b4d60 2024-03-25T19:40:17.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.415+0000 7f654dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.415+0000 7f654dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.415+0000 7f654dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.415+0000 7f654dffb640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:40:17.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.415+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6538003520 con 0x7f65500b4d60 2024-03-25T19:40:17.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.419+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633989239 0 0) 0x7f65500af940 con 0x7f65500b1730 2024-03-25T19:40:17.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.419+0000 7f654dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.419+0000 7f654dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.419+0000 7f654dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.419+0000 7f654dffb640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:40:17.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.419+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65500b0a70 con 0x7f65500b1730 2024-03-25T19:40:17.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.429+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290662832 0 0) 0x7f65500b0d50 con 0x7f655014be90 2024-03-25T19:40:17.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.429+0000 7f654dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.429+0000 7f654dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.429+0000 7f654dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.429+0000 7f654dffb640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:40:17.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.429+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65500af940 con 0x7f655014be90 2024-03-25T19:40:17.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.430+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 <== mon.1 v1:172.21.15.188:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3479581485 0 0) 0x7f6538003520 con 0x7f65500b4d60 2024-03-25T19:40:17.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.430+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 >> v1:172.21.15.19:6790/0 conn(0x7f655014be90 legacy=0x7f655014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.430+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 >> v1:172.21.15.19:6789/0 conn(0x7f65500b1730 legacy=0x7f65500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.430+0000 7f654dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:40:17.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.430+0000 7f654dffb640 1 monclient: found mon.noname-c 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.430+0000 7f654dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.430+0000 7f654dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.188:6789/0 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.430+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 --> v1:172.21.15.188:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65500b1030 con 0x7f65500b4d60 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.430+0000 7f654dffb640 10 monclient: _finish_auth 0 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.430+0000 7f654dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.430+0000 7f654dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.431+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 <== mon.1 v1:172.21.15.188:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f6540001690 con 0x7f65500b4d60 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.431+0000 7f654dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.431+0000 7f654dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.431+0000 7f654dffb640 10 monclient: dump: 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.432 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:17.433 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:17.433 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:17.433 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:17.433 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:17.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.431+0000 7f65559cf640 5 monclient: authenticate success, global_id 4726 2024-03-25T19:40:17.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.431+0000 7f65559cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:17.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.445+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 <== mon.1 v1:172.21.15.188:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65400017f0 con 0x7f65500b4d60 2024-03-25T19:40:17.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.446+0000 7f654dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.446+0000 7f654dffb640 1 -- 172.21.15.19:0/3417303966 <== mon.1 v1:172.21.15.188:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f6540001c00 con 0x7f65500b4d60 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.446+0000 7f654dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.446+0000 7f654dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.446+0000 7f654dffb640 10 monclient: dump: 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.446+0000 7f65559cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:17.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.446+0000 7f65559cf640 4 set_mon_vals no callback set 2024-03-25T19:40:17.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.446+0000 7f65559cf640 10 monclient: shutdown 2024-03-25T19:40:17.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.446+0000 7f65559cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:17.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.446+0000 7f65559cf640 1 -- 172.21.15.19:0/3417303966 >> v1:172.21.15.188:6789/0 conn(0x7f65500b4d60 legacy=0x7f655013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.446+0000 7f65559cf640 1 -- 172.21.15.19:0/3417303966 shutdown_connections 2024-03-25T19:40:17.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.446+0000 7f65559cf640 1 -- 172.21.15.19:0/3417303966 >> 172.21.15.19:0/3417303966 conn(0x7f65500617a0 msgr2=0x7f6550061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:17.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.447+0000 7f65559cf640 1 -- 172.21.15.19:0/3417303966 shutdown_connections 2024-03-25T19:40:17.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.447+0000 7f65559cf640 1 -- 172.21.15.19:0/3417303966 wait complete. 2024-03-25T19:40:17.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.447+0000 7f65559cf640 10 monclient: build_initial_monmap 2024-03-25T19:40:17.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.447+0000 7f65559cf640 10 monclient: monmap: 2024-03-25T19:40:17.448 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:40:17.448 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:17.448 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.448+0000 7f65559cf640 1 librados: starting msgr at 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.448+0000 7f65559cf640 1 librados: starting objecter 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.448+0000 7f65559cf640 1 Processor -- start 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.448+0000 7f65559cf640 1 -- start start 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.448+0000 7f65559cf640 1 librados: setting wanted keys 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.448+0000 7f65559cf640 1 librados: calling monclient init 2024-03-25T19:40:17.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.448+0000 7f65559cf640 10 monclient: init 2024-03-25T19:40:17.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.449+0000 7f65559cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:17.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.449+0000 7f65559cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:17.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.449+0000 7f65559cf640 10 monclient(hunting): picked mon.noname-c con 0x7f65500b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:40:17.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.449+0000 7f65559cf640 10 monclient(hunting): picked mon.noname-a con 0x7f65500b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:40:17.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.449+0000 7f65559cf640 10 monclient(hunting): picked mon.noname-b con 0x7f655014be90 addr v1:172.21.15.188:6789/0 2024-03-25T19:40:17.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.449+0000 7f65559cf640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6550135810 con 0x7f65500b4d60 2024-03-25T19:40:17.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.449+0000 7f65559cf640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65501dac70 con 0x7f655014be90 2024-03-25T19:40:17.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.449+0000 7f65559cf640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65501daf50 con 0x7f65500b1730 2024-03-25T19:40:17.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.449+0000 7f65559cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:17.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.449+0000 7f65559cf640 10 monclient(hunting): authenticate will time out at 6670.719342s 2024-03-25T19:40:17.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.463+0000 7f654effd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f65500b1730 0x7f65501db450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:35118/0 (socket says 172.21.15.19:35118) 2024-03-25T19:40:17.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.463+0000 7f654effd640 1 -- 172.21.15.19:0/382487707 learned_addr learned my addr 172.21.15.19:0/382487707 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:40:17.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.464+0000 7f6537fff640 10 client.?.objecter ms_handle_connect 0x7f65500b1730 2024-03-25T19:40:17.464 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.464+0000 7f6537fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:17.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.472+0000 7f6537fff640 10 client.?.objecter ms_handle_connect 0x7f65500b4d60 2024-03-25T19:40:17.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.472+0000 7f6537fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:17.476 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.476+0000 7f6537fff640 10 client.?.objecter ms_handle_connect 0x7f655014be90 2024-03-25T19:40:17.476 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.476+0000 7f6537fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:17.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.478+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587874989 0 0) 0x7f65501daf50 con 0x7f65500b1730 2024-03-25T19:40:17.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.478+0000 7f6537fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.478+0000 7f6537fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.478+0000 7f6537fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.478+0000 7f6537fff640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:40:17.478 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.478+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f653c004eb0 con 0x7f65500b1730 2024-03-25T19:40:17.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.484+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861817113 0 0) 0x7f6550135810 con 0x7f65500b4d60 2024-03-25T19:40:17.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.484+0000 7f6537fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.484 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.484+0000 7f6537fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.484+0000 7f6537fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.484+0000 7f6537fff640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:40:17.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.484+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65501daf50 con 0x7f65500b4d60 2024-03-25T19:40:17.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.489+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812479945 0 0) 0x7f65501dac70 con 0x7f655014be90 2024-03-25T19:40:17.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.489+0000 7f6537fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:17.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.489+0000 7f6537fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:17.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.489+0000 7f6537fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:17.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.489+0000 7f6537fff640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:40:17.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.489+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6550135810 con 0x7f655014be90 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.493+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4120452964 0 0) 0x7f653c004eb0 con 0x7f65500b1730 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.493+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65501dac70 con 0x7f65500b1730 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.493+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f6540001bc0 con 0x7f65500b1730 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.493+0000 7f6537fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.493+0000 7f6537fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.493+0000 7f6537fff640 10 monclient(hunting): dump: 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.493 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:17.494 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:17.494 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:17.494 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:17.494 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:17.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.493+0000 7f6537fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:17.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.500+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 394060891 0 0) 0x7f65501daf50 con 0x7f65500b4d60 2024-03-25T19:40:17.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.500+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f653c004eb0 con 0x7f65500b4d60 2024-03-25T19:40:17.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.500+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f6544001970 con 0x7f65500b4d60 2024-03-25T19:40:17.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.500+0000 7f6537fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.500+0000 7f6537fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:17.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.500+0000 7f6537fff640 10 monclient(hunting): dump: 2024-03-25T19:40:17.500 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:17.500 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:17.500 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:17.500 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:17.500 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.501 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:17.501 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:17.501 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:17.501 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:17.501 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2564101444 0 0) 0x7f65501dac70 con 0x7f65500b1730 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 >> v1:172.21.15.188:6789/0 conn(0x7f655014be90 legacy=0x7f6550135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 >> v1:172.21.15.19:6789/0 conn(0x7f65500b4d60 legacy=0x7f65501349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f6537fff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f6537fff640 1 monclient: found mon.c 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f6537fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f6537fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65501dbb60 con 0x7f65500b1730 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f6537fff640 10 monclient: _finish_auth 0 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f6537fff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f6537fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f65559cf640 5 monclient: authenticate success, global_id 4706 2024-03-25T19:40:17.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f65559cf640 10 monclient: _renew_subs 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f65559cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f65559cf640 1 -- 172.21.15.19:0/382487707 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655013b1b0 con 0x7f65500b1730 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f65559cf640 10 client.4706.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f65559cf640 10 monclient: _renew_subs 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f65559cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f65559cf640 1 -- 172.21.15.19:0/382487707 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f655013bd30 con 0x7f65500b1730 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.505+0000 7f65559cf640 1 librados: init done 2024-03-25T19:40:17.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.507+0000 7f65559cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:17.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.507+0000 7f65559cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:17.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.507+0000 7f65559cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:40:17.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.507+0000 7f65559cf640 1 -- 172.21.15.19:0/382487707 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65500627b0 con 0x7f65500b1730 2024-03-25T19:40:17.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.848+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65400011f0 con 0x7f65500b1730 2024-03-25T19:40:17.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.848+0000 7f6537fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:17.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.848+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f6540003b00 con 0x7f65500b1730 2024-03-25T19:40:17.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.848+0000 7f6537fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:17.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.848+0000 7f6537fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:17.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.848+0000 7f6537fff640 10 monclient: dump: 2024-03-25T19:40:17.848 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:17.848 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:17.849 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:17.849 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:17.849 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:17.849 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:17.849 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:17.849 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:17.849 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:17.849 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:17.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.848+0000 7f654d7fa640 4 set_mon_vals no callback set 2024-03-25T19:40:17.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.849+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f6540003d20 con 0x7f65500b1730 2024-03-25T19:40:17.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.849+0000 7f654e7fc640 10 monclient: get_auth_request con 0x7f653c06df70 auth_method 0 2024-03-25T19:40:17.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.850+0000 7f6537fff640 10 client.4706.objecter ms_handle_connect 0x7f653c06df70 2024-03-25T19:40:17.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.857+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1490778728 0 0) 0x7f6540054880 con 0x7f65500b1730 2024-03-25T19:40:17.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.857+0000 7f6537fff640 10 client.4706.objecter ms_dispatch 0x7f6550000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:40:17.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.857+0000 7f6537fff640 3 client.4706.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:40:17.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.857+0000 7f6537fff640 3 client.4706.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:40:17.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.857+0000 7f6537fff640 20 client.4706.objecter dump_active .. 0 homeless 2024-03-25T19:40:17.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.857+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6540050390 con 0x7f65500b1730 2024-03-25T19:40:17.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.857+0000 7f6537fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:17.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:17.857+0000 7f6537fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:18.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.116+0000 7f65559cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:18.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.116+0000 7f65559cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:18.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.116+0000 7f65559cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-25T19:40:18.117 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.116+0000 7f65559cf640 1 -- 172.21.15.19:0/382487707 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6550069f40 con 0x7f65500b1730 2024-03-25T19:40:18.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.224+0000 7f6537fff640 1 -- 172.21.15.19:0/382487707 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1576595288) 0x7f65400522c0 con 0x7f65500b1730 2024-03-25T19:40:18.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.224+0000 7f6537fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:18.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.224+0000 7f6537fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:18.224 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:40:18.224 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:38:58.470042+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:58.470042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:18.226 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.226+0000 7f65559cf640 10 librados: watch_flush enter 2024-03-25T19:40:18.226 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.226+0000 7f65559cf640 10 librados: watch_flush exit 2024-03-25T19:40:18.226 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.226+0000 7f65559cf640 20 client.4706.objecter shutdown clearing up homeless session... 2024-03-25T19:40:18.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.226+0000 7f65559cf640 10 client.4706.objecter successfully canceled tick 2024-03-25T19:40:18.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.226+0000 7f65559cf640 1 -- 172.21.15.19:0/382487707 >> v1:172.21.15.188:6800/2207506229 conn(0x7f653c06df70 legacy=0x7f653c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:18.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.226+0000 7f65559cf640 10 monclient: shutdown 2024-03-25T19:40:18.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.226+0000 7f65559cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:18.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.226+0000 7f65559cf640 1 -- 172.21.15.19:0/382487707 >> v1:172.21.15.19:6790/0 conn(0x7f65500b1730 legacy=0x7f65501db450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:18.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.226+0000 7f65559cf640 1 -- 172.21.15.19:0/382487707 shutdown_connections 2024-03-25T19:40:18.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.226+0000 7f65559cf640 1 -- 172.21.15.19:0/382487707 >> 172.21.15.19:0/382487707 conn(0x7f65500617a0 msgr2=0x7f65500b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:18.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.226+0000 7f65559cf640 1 -- 172.21.15.19:0/382487707 shutdown_connections 2024-03-25T19:40:18.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.226+0000 7f65559cf640 1 -- 172.21.15.19:0/382487707 wait complete. 2024-03-25T19:40:18.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:18.227+0000 7f65559cf640 1 librados: shutdown 2024-03-25T19:40:18.238 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:18.816 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:41.545 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.825 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:41.555 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.867 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:41.596 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.892 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:41.621 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.895 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:41.624 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.904 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:11:41.607 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.918 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:11:41.621 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.928 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:41.658 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:18.952 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:11:41.655 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:25.045 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:25.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.146+0000 7f22b078f640 10 monclient: get_monmap_and_config 2024-03-25T19:40:25.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.146+0000 7f22b078f640 10 monclient: build_initial_monmap 2024-03-25T19:40:25.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.146+0000 7f22b078f640 10 monclient: monmap: 2024-03-25T19:40:25.147 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:40:25.147 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:25.147 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:40:25.147916+0000 2024-03-25T19:40:25.147 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:40:25.147916+0000 2024-03-25T19:40:25.147 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:25.147 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:25.147 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:40:25.147 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-c 2024-03-25T19:40:25.147 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-25T19:40:25.147 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.147+0000 7f22b078f640 1 Processor -- start 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.147+0000 7f22b078f640 1 -- start start 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.147+0000 7f22b078f640 10 monclient: init 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.148+0000 7f22b078f640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.148+0000 7f22b078f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.148+0000 7f22b078f640 10 monclient(hunting): picked mon.noname-a con 0x7f22a80b4d60 addr v1:172.21.15.19:6789/0 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.148+0000 7f22b078f640 10 monclient(hunting): picked mon.noname-c con 0x7f22a80b1730 addr v1:172.21.15.188:6789/0 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.148+0000 7f22b078f640 10 monclient(hunting): picked mon.noname-b con 0x7f22a814be90 addr v1:172.21.15.19:6790/0 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.148+0000 7f22b078f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80af940 con 0x7f22a80b4d60 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.148+0000 7f22b078f640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80b0a70 con 0x7f22a80b1730 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.148+0000 7f22b078f640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a80b0d50 con 0x7f22a814be90 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.148+0000 7f22b078f640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:25.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.148+0000 7f22b078f640 10 monclient(hunting): authenticate will time out at 6678.418642s 2024-03-25T19:40:25.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.166+0000 7f22ae52b640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f22a80b4d60 0x7f22a813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:57290/0 (socket says 172.21.15.19:57290) 2024-03-25T19:40:25.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.166+0000 7f22ae52b640 1 -- 172.21.15.19:0/1142350520 learned_addr learned my addr 172.21.15.19:0/1142350520 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:40:25.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.174+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097630663 0 0) 0x7f22a80af940 con 0x7f22a80b4d60 2024-03-25T19:40:25.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.174+0000 7f22ad529640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.174+0000 7f22ad529640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.174+0000 7f22ad529640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.174+0000 7f22ad529640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:40:25.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.174+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2294003520 con 0x7f22a80b4d60 2024-03-25T19:40:25.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.181+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234034798 0 0) 0x7f22a80b0a70 con 0x7f22a80b1730 2024-03-25T19:40:25.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.181+0000 7f22ad529640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.181+0000 7f22ad529640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.181+0000 7f22ad529640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.181+0000 7f22ad529640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:40:25.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.181+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a80af940 con 0x7f22a80b1730 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.189+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308778297 0 0) 0x7f22a80b0d50 con 0x7f22a814be90 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.189+0000 7f22ad529640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.189+0000 7f22ad529640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.189+0000 7f22ad529640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.189+0000 7f22ad529640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:40:25.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.189+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a80b0a70 con 0x7f22a814be90 2024-03-25T19:40:25.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2680092766 0 0) 0x7f2294003520 con 0x7f22a80b4d60 2024-03-25T19:40:25.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 >> v1:172.21.15.19:6790/0 conn(0x7f22a814be90 legacy=0x7f22a814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 >> v1:172.21.15.188:6789/0 conn(0x7f22a80b1730 legacy=0x7f22a80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 10 monclient: _finish_hunting 0 2024-03-25T19:40:25.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 1 monclient: found mon.noname-a 2024-03-25T19:40:25.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:25.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-25T19:40:25.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a80b1030 con 0x7f22a80b4d60 2024-03-25T19:40:25.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 10 monclient: _finish_auth 0 2024-03-25T19:40:25.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 10 monclient: _check_auth_tickets 2024-03-25T19:40:25.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:25.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f22a4001690 con 0x7f22a80b4d60 2024-03-25T19:40:25.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22b078f640 5 monclient: authenticate success, global_id 4734 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22b078f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22ad529640 10 monclient: dump: 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:25.196 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.194+0000 7f22b078f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.217+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22a40017f0 con 0x7f22a80b4d60 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.217+0000 7f22ad529640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.218+0000 7f22ad529640 1 -- 172.21.15.19:0/1142350520 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f22a4001c00 con 0x7f22a80b4d60 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.218+0000 7f22ad529640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.218+0000 7f22ad529640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.218+0000 7f22ad529640 10 monclient: dump: 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.218 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:25.219 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:25.219 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:25.219 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:25.219 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:25.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.218+0000 7f22b078f640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:25.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.218+0000 7f22b078f640 4 set_mon_vals no callback set 2024-03-25T19:40:25.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.218+0000 7f22b078f640 10 monclient: shutdown 2024-03-25T19:40:25.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.218+0000 7f22b078f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.218+0000 7f22b078f640 1 -- 172.21.15.19:0/1142350520 >> v1:172.21.15.19:6789/0 conn(0x7f22a80b4d60 legacy=0x7f22a813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.219+0000 7f22b078f640 1 -- 172.21.15.19:0/1142350520 shutdown_connections 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.219+0000 7f22b078f640 1 -- 172.21.15.19:0/1142350520 >> 172.21.15.19:0/1142350520 conn(0x7f22a80617a0 msgr2=0x7f22a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.219+0000 7f22b078f640 1 -- 172.21.15.19:0/1142350520 shutdown_connections 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.219+0000 7f22b078f640 1 -- 172.21.15.19:0/1142350520 wait complete. 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.220+0000 7f22b078f640 10 monclient: build_initial_monmap 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.220+0000 7f22b078f640 10 monclient: monmap: 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:25.222 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.noname-b 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.220+0000 7f22b078f640 1 librados: starting msgr at 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.220+0000 7f22b078f640 1 librados: starting objecter 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.220+0000 7f22b078f640 1 Processor -- start 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.220+0000 7f22b078f640 1 -- start start 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.220+0000 7f22b078f640 1 librados: setting wanted keys 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.220+0000 7f22b078f640 1 librados: calling monclient init 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.220+0000 7f22b078f640 10 monclient: init 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.221+0000 7f22b078f640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.221+0000 7f22b078f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.221+0000 7f22b078f640 10 monclient(hunting): picked mon.noname-c con 0x7f22a80b1730 addr v1:172.21.15.19:6790/0 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.221+0000 7f22b078f640 10 monclient(hunting): picked mon.noname-b con 0x7f22a80b4d60 addr v1:172.21.15.188:6789/0 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.221+0000 7f22b078f640 10 monclient(hunting): picked mon.noname-a con 0x7f22a814be90 addr v1:172.21.15.19:6789/0 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.221+0000 7f22b078f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a81d87b0 con 0x7f22a814be90 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.221+0000 7f22b078f640 1 -- --> v1:172.21.15.188:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a81388e0 con 0x7f22a80b4d60 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.221+0000 7f22b078f640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a81d78a0 con 0x7f22a80b1730 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.221+0000 7f22b078f640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.221+0000 7f22b078f640 10 monclient(hunting): authenticate will time out at 6678.491363s 2024-03-25T19:40:25.225 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.226+0000 7f22ae52b640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f22a80b1730 0x7f22a81d80a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:54216/0 (socket says 172.21.15.19:54216) 2024-03-25T19:40:25.226 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.226+0000 7f22ae52b640 1 -- 172.21.15.19:0/2080178136 learned_addr learned my addr 172.21.15.19:0/2080178136 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-25T19:40:25.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.229+0000 7f229effd640 10 client.?.objecter ms_handle_connect 0x7f22a80b4d60 2024-03-25T19:40:25.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.229+0000 7f229effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:25.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.231+0000 7f229effd640 10 client.?.objecter ms_handle_connect 0x7f22a80b1730 2024-03-25T19:40:25.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.231+0000 7f229effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:25.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.231+0000 7f229effd640 10 client.?.objecter ms_handle_connect 0x7f22a814be90 2024-03-25T19:40:25.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.231+0000 7f229effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:25.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.238+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142563623 0 0) 0x7f22a81d87b0 con 0x7f22a814be90 2024-03-25T19:40:25.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.238+0000 7f229effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.238+0000 7f229effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.238+0000 7f229effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.238+0000 7f229effd640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:40:25.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.238+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c004eb0 con 0x7f22a814be90 2024-03-25T19:40:25.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.241+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.1 v1:172.21.15.188:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612663083 0 0) 0x7f22a81388e0 con 0x7f22a80b4d60 2024-03-25T19:40:25.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.241+0000 7f229effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.241+0000 7f229effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.241+0000 7f229effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.241+0000 7f229effd640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:40:25.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.241+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 --> v1:172.21.15.188:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a81d87b0 con 0x7f22a80b4d60 2024-03-25T19:40:25.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.245+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781778691 0 0) 0x7f22a81d78a0 con 0x7f22a80b1730 2024-03-25T19:40:25.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.245+0000 7f229effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:25.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.245+0000 7f229effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:25.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.245+0000 7f229effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:25.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.245+0000 7f229effd640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:40:25.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.245+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a81388e0 con 0x7f22a80b1730 2024-03-25T19:40:25.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.256+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4105109340 0 0) 0x7f22a81388e0 con 0x7f22a80b1730 2024-03-25T19:40:25.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.256+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22a81d78a0 con 0x7f22a80b1730 2024-03-25T19:40:25.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.256+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f22a4001b00 con 0x7f22a80b1730 2024-03-25T19:40:25.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.256+0000 7f229effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.256+0000 7f229effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:25.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.256+0000 7f229effd640 10 monclient(hunting): dump: 2024-03-25T19:40:25.257 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:25.257 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:25.257 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:25.257 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:25.257 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.257 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:25.257 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:25.257 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:25.257 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:25.257 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:25.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.256+0000 7f229effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:25.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.258+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1027335121 0 0) 0x7f228c004eb0 con 0x7f22a814be90 2024-03-25T19:40:25.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.258+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22a81388e0 con 0x7f22a814be90 2024-03-25T19:40:25.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.258+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f22a0001bd0 con 0x7f22a814be90 2024-03-25T19:40:25.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.258+0000 7f229effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.258+0000 7f229effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:25.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.258+0000 7f229effd640 10 monclient(hunting): dump: 2024-03-25T19:40:25.258 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:25.258 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:25.258 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:25.259 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:25.259 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.259 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:25.259 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:25.259 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:25.259 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:25.259 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:25.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2668307882 0 0) 0x7f22a81388e0 con 0x7f22a814be90 2024-03-25T19:40:25.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 >> v1:172.21.15.19:6790/0 conn(0x7f22a80b1730 legacy=0x7f22a81d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 >> v1:172.21.15.188:6789/0 conn(0x7f22a80b4d60 legacy=0x7f22a8136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f229effd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:25.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f229effd640 1 monclient: found mon.a 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f229effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f229effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a81d7b80 con 0x7f22a814be90 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f229effd640 10 monclient: _finish_auth 0 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f229effd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f229effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f22b078f640 5 monclient: authenticate success, global_id 4737 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f22b078f640 10 monclient: _renew_subs 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f22b078f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f22b078f640 1 -- 172.21.15.19:0/2080178136 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a813bc60 con 0x7f22a814be90 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f22b078f640 10 client.4737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f22b078f640 10 monclient: _renew_subs 2024-03-25T19:40:25.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f22b078f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:40:25.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f22b078f640 1 -- 172.21.15.19:0/2080178136 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a8137570 con 0x7f22a814be90 2024-03-25T19:40:25.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.263+0000 7f22b078f640 1 librados: init done 2024-03-25T19:40:25.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.264+0000 7f22b078f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:25.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.264+0000 7f22b078f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:25.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.264+0000 7f22b078f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:40:25.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.264+0000 7f22b078f640 1 -- 172.21.15.19:0/2080178136 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a80627b0 con 0x7f22a814be90 2024-03-25T19:40:25.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.585+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22a0002060 con 0x7f22a814be90 2024-03-25T19:40:25.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.585+0000 7f229effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:25.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.585+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4266820288 0 0) 0x7f22a0003b30 con 0x7f22a814be90 2024-03-25T19:40:25.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.585+0000 7f229effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:25.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.585+0000 7f229effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:25.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.585+0000 7f229effd640 10 monclient: dump: 2024-03-25T19:40:25.585 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-25T19:40:25.585 INFO:teuthology.misc.health.smithi019.stderr:fsid 6efd308f-8e81-46e0-802b-9ba73517b91b 2024-03-25T19:40:25.585 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:25.585 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-25T19:28:04.382619+0000 2024-03-25T19:40:25.585 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:25.586 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 3 2024-03-25T19:40:25.586 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-25T19:40:25.586 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.188:6789/0 mon.b 2024-03-25T19:40:25.586 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-25T19:40:25.586 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-25T19:40:25.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.585+0000 7f22acd28640 4 set_mon_vals no callback set 2024-03-25T19:40:25.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.585+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4113818411 0 0) 0x7f22a0003d50 con 0x7f22a814be90 2024-03-25T19:40:25.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.586+0000 7f22ae52b640 10 monclient: get_auth_request con 0x7f228c06df70 auth_method 0 2024-03-25T19:40:25.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.586+0000 7f229effd640 10 client.4737.objecter ms_handle_connect 0x7f228c06df70 2024-03-25T19:40:25.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.592+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1490778728 0 0) 0x7f22a0053820 con 0x7f22a814be90 2024-03-25T19:40:25.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.592+0000 7f229effd640 10 client.4737.objecter ms_dispatch 0x7f22a8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:40:25.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.592+0000 7f229effd640 3 client.4737.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:40:25.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.592+0000 7f229effd640 3 client.4737.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:40:25.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.592+0000 7f229effd640 20 client.4737.objecter dump_active .. 0 homeless 2024-03-25T19:40:25.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.592+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22a004f330 con 0x7f22a814be90 2024-03-25T19:40:25.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.592+0000 7f229effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:25.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.592+0000 7f229effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:25.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.852+0000 7f22b078f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:25.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.852+0000 7f22b078f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:25.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.852+0000 7f22b078f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-25T19:40:25.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.852+0000 7f22b078f640 1 -- 172.21.15.19:0/2080178136 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22a81378c0 con 0x7f22a814be90 2024-03-25T19:40:25.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.876+0000 7f229effd640 1 -- 172.21.15.19:0/2080178136 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1576595288) 0x7f22a0051260 con 0x7f22a814be90 2024-03-25T19:40:25.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.876+0000 7f229effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:25.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.876+0000 7f229effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:25.876 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-25T19:40:25.877 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"6efd308f-8e81-46e0-802b-9ba73517b91b","created":"2024-03-25T19:28:54.750516+0000","modified":"2024-03-25T19:38:58.470042+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:38:58.470042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c96228b-453c-4e10-85b0-50a616f9e06b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"52a1fe90-7f0c-443d-b654-68ad85b9d9ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a8b5725b-de23-4b36-83b4-4276ca95aad7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"8f6cea59-df77-45e1-960e-96fcfe4f78b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e72f23f5-3b0d-4046-80d1-074992d80443","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcee913e-5189-4fcd-b37e-2a62542e6c3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"eb7504bd-4506-4bdb-b58a-260851ce16b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3e4577eb-444c-4860-afd9-f52b1b2f00d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 10 librados: watch_flush enter 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 10 librados: watch_flush exit 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 20 client.4737.objecter shutdown clearing up homeless session... 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 10 client.4737.objecter successfully canceled tick 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 1 -- 172.21.15.19:0/2080178136 >> v1:172.21.15.188:6800/2207506229 conn(0x7f228c06df70 legacy=0x7f228c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 10 monclient: shutdown 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 1 -- 172.21.15.19:0/2080178136 >> v1:172.21.15.19:6789/0 conn(0x7f22a814be90 legacy=0x7f22a813b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 1 -- 172.21.15.19:0/2080178136 shutdown_connections 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 1 -- 172.21.15.19:0/2080178136 >> 172.21.15.19:0/2080178136 conn(0x7f22a80617a0 msgr2=0x7f22a80b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 1 -- 172.21.15.19:0/2080178136 shutdown_connections 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 1 -- 172.21.15.19:0/2080178136 wait complete. 2024-03-25T19:40:25.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-25T19:40:25.879+0000 7f22b078f640 1 librados: shutdown 2024-03-25T19:40:25.893 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:25.893 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:40:25.896 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:40:25.896 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:40:25.897 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:27.799 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:11:26.784 36133== 2024-03-25T19:40:27.799 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:11:26.784 36133== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:27.837 INFO:tasks.ceph.osd.0.smithi019.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:31.999 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:40:32.000 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:40:32.000 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:40:32.001 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:34.049 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:11:33.025 36135== 2024-03-25T19:40:34.049 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:11:33.025 36135== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:34.084 INFO:tasks.ceph.osd.1.smithi019.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:38.102 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:38.102 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:40:38.103 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:40:38.103 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:40:38.103 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:38.816 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:01.545 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.865 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:01.595 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.895 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:01.624 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.918 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:12:01.622 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.928 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:01.658 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.931 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:12:01.634 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.945 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:01.674 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.953 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:12:01.656 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:38.969 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:01.699 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:39.884 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:11:38.872 36131== 2024-03-25T19:40:39.884 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:11:38.872 36131== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:39.926 INFO:tasks.ceph.osd.2.smithi019.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:44.205 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:40:44.205 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:40:44.206 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:40:44.206 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:46.113 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:11:45.086 36137== 2024-03-25T19:40:46.113 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:11:45.086 36137== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:46.152 INFO:tasks.ceph.osd.3.smithi019.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:50.307 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:40:50.307 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:40:50.308 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:40:50.308 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:52.228 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:11:51.192 35606== 2024-03-25T19:40:52.228 INFO:tasks.ceph.osd.4.smithi188.stderr:==00:00:11:51.192 35606== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:52.265 INFO:tasks.ceph.osd.4.smithi188.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:56.409 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi188 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:40:56.410 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:40:56.410 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:40:56.410 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:40:58.211 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:11:57.167 35613== 2024-03-25T19:40:58.212 INFO:tasks.ceph.osd.5.smithi188.stderr:==00:00:11:57.167 35613== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:40:58.245 INFO:tasks.ceph.osd.5.smithi188.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:59.045 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:21.774 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.092 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:21.822 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.092 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:12:21.795 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.094 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:21.823 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.094 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:12:21.798 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.096 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:21.825 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.118 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:12:21.821 35008== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.121 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:21.850 35825== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:59.123 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:21.853 35823== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:41:02.512 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi188 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:41:02.513 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:41:02.513 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:41:02.513 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:04.316 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:12:03.272 35614== 2024-03-25T19:41:04.316 INFO:tasks.ceph.osd.6.smithi188.stderr:==00:00:12:03.272 35614== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:41:04.360 INFO:tasks.ceph.osd.6.smithi188.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:41:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:41:08.615 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi188 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:41:08.616 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:41:08.616 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:41:08.616 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:10.181 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:12:09.123 35615== 2024-03-25T19:41:10.181 INFO:tasks.ceph.osd.7.smithi188.stderr:==00:00:12:09.123 35615== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:41:10.210 INFO:tasks.ceph.osd.7.smithi188.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:41:14.718 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:41:14.718 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi188 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:41:14.719 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:41:14.719 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:41:14.719 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:41:14.719 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:14.746 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:41:14.746 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:41:14.747 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:14.747 INFO:tasks.ceph.mgr.x.smithi188.stderr:2024-03-25T19:41:14.748+0000 7fcf0a351640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 35006) UID: 0 2024-03-25T19:41:14.774 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:41:14.774 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:41:14.775 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:41:14.775 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:14.807 INFO:tasks.ceph.mon.a.smithi019.stderr:2024-03-25T19:41:14.792+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35790) UID: 0 2024-03-25T19:41:14.807 INFO:tasks.ceph.mon.a.smithi019.stderr:2024-03-25T19:41:14.793+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:41:15.421 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:38.150 35823== 2024-03-25T19:41:15.421 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:38.150 35823== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:41:15.430 INFO:tasks.ceph.mon.a.smithi019.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:41:20.876 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:41:20.876 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:41:20.876 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:41:20.877 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:41:20.877 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:20.893 INFO:tasks.ceph.mon.c.smithi019.stderr:2024-03-25T19:41:20.882+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35792) UID: 0 2024-03-25T19:41:20.893 INFO:tasks.ceph.mon.c.smithi019.stderr:2024-03-25T19:41:20.883+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:41:21.419 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:44.149 35825== 2024-03-25T19:41:21.419 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:44.149 35825== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:41:21.428 INFO:tasks.ceph.mon.c.smithi019.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:41:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:41:26.979 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:41:26.979 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:41:26.979 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:41:26.980 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:26.995 INFO:tasks.ceph.mon.b.smithi188.stderr:2024-03-25T19:41:26.984+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35004) UID: 0 2024-03-25T19:41:26.995 INFO:tasks.ceph.mon.b.smithi188.stderr:2024-03-25T19:41:26.986+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T19:41:27.566 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:12:50.269 35008== 2024-03-25T19:41:27.566 INFO:tasks.ceph.mon.b.smithi188.stderr:==00:00:12:50.269 35008== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:41:27.576 INFO:tasks.ceph.mon.b.smithi188.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:41:33.081 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:41:33.081 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi188 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:41:33.081 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:41:33.082 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:41:33.082 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:41:33.130 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-25T19:41:33.130 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:41:33.231 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-25T19:41:33.232 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:41:33.289 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-25T19:41:33.289 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:41:33.345 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-25T19:41:33.345 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:41:33.396 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi188.front.sepia.ceph.com 2024-03-25T19:41:33.396 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:41:33.532 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi188.front.sepia.ceph.com 2024-03-25T19:41:33.533 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:41:33.622 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi188.front.sepia.ceph.com 2024-03-25T19:41:33.622 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:41:33.679 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi188.front.sepia.ceph.com 2024-03-25T19:41:33.679 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:41:33.728 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:41:33.729 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310/data/mon.a.tgz 2024-03-25T19:41:33.729 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-03-25T19:41:33.744 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.CYLBz663eI 2024-03-25T19:41:33.745 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.CYLBz663eI 2024-03-25T19:41:33.850 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.CYLBz663eI 2024-03-25T19:41:33.933 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.CYLBz663eI is 133KB 2024-03-25T19:41:33.985 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.CYLBz663eI 2024-03-25T19:41:33.999 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310/data/mon.c.tgz 2024-03-25T19:41:33.999 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-03-25T19:41:34.055 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.3nfj4ZQrTD 2024-03-25T19:41:34.056 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.3nfj4ZQrTD 2024-03-25T19:41:34.143 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.3nfj4ZQrTD 2024-03-25T19:41:34.227 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.3nfj4ZQrTD is 133KB 2024-03-25T19:41:34.278 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.3nfj4ZQrTD 2024-03-25T19:41:34.292 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310/data/mon.b.tgz 2024-03-25T19:41:34.292 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-03-25T19:41:34.309 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.oJ5MUh7QkV 2024-03-25T19:41:34.310 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.oJ5MUh7QkV 2024-03-25T19:41:34.413 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.oJ5MUh7QkV 2024-03-25T19:41:34.501 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.oJ5MUh7QkV is 133KB 2024-03-25T19:41:34.524 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.oJ5MUh7QkV 2024-03-25T19:41:34.539 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:41:34.540 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:41:34.542 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:41:34.924 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:41:35.144 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:41:35.146 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310/remote/smithi019/crash 2024-03-25T19:41:35.147 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:41:35.173 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310/remote/smithi188/crash 2024-03-25T19:41:35.174 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:41:35.201 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:41:35.201 DEBUG:teuthology.orchestra.run.smithi019:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:41:35.218 DEBUG:teuthology.orchestra.run.smithi188:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:41:35.241 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:41:35.242 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40531.log 2024-03-25T19:41:35.294 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:41:35.294 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr: 75.9% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/valgrind/osd.6.log: /var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35362.log 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.35362.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:41:35.295 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35407.log 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35362.log.gz 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35202.log 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.35407.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.35202.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35407.log.gz 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.7.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35202.log.gz 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35453.log 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35322.log 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.35453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35242.log 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.35322.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35162.log 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35453.log.gz 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.35242.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35322.log.gz 2024-03-25T19:41:35.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-client.admin.35162.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.35242.log.gz 2024-03-25T19:41:35.297 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35282.log 2024-03-25T19:41:35.297 INFO:teuthology.orchestra.run.smithi188.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.35162.log.gz 2024-03-25T19:41:35.297 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:41:35.297 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.35282.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:41:35.297 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35282.log.gz 2024-03-25T19:41:35.297 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39905.log 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.40531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39708.log 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39905.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40531.log.gz 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40375.log 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41975.log 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39905.log.gz 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39708.log: /var/log/ceph/ceph-client.admin.40375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42833.log 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41194.log 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40375.log.gz 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39708.log.gz 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41819.log 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41975.log.gz 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42599.log 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42833.log.gz 2024-03-25T19:41:35.300 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40843.log 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41194.log.gz 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40453.log 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41819.log.gz 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41428.log 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42599.log.gz 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36161.log 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40453.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40843.log.gz 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39827.log 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40453.log.gz 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.36161.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41428.log.gz 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42872.log 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40219.log 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36161.log.gz 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42872.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39827.log.gz 2024-03-25T19:41:35.301 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40219.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42872.log.gz 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/valgrind/mon.c.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40219.log.gz 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr: 80.8% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/valgrind/mon.a.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40492.log 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42443.log 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:41:35.302 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42560.log 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42911.log 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40492.log.gz 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39669.log 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42443.log.gz 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42911.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42560.log.gz 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42794.log 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41077.log 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42911.log.gz 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42794.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39669.log.gz 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41702.log 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42131.log 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42794.log.gz 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41897.log 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41077.log.gz 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40687.log 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41702.log.gz 2024-03-25T19:41:35.303 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41038.log 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42131.log.gz 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40804.log 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41897.log.gz 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41038.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40687.log.gz 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42482.log 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39944.log 86.8% 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41038.log.gz 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40726.log 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40804.log.gz 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39985.log 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42482.log.gz 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39944.log.gz 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41155.log 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42209.log 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41780.log 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40726.log.gz 2024-03-25T19:41:35.304 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39985.log.gz 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42209.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41155.log.gz 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42092.log 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42209.log.gz 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39749.log 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41780.log.gz 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41272.log 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39749.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42092.log.gz 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41663.log 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41936.log 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39749.log.gz 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35832.log 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41272.log.gz 2024-03-25T19:41:35.305 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42638.log 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41663.log.gz 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.35832.log: 86.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.41936.log.gz 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41545.log 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39464.log 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35832.log.gz 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41233.log 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42638.log.gz 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42950.log 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41545.log.gz 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40882.log 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39464.log.gz 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37232.log 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41233.log.gz 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42404.log 2024-03-25T19:41:35.306 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42950.log.gz 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37232.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29967.log 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40882.log.gz 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42014.log 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37232.log.gz 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.tmp-client.admin.29967.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29967.log.gz 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42404.log.gz 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40921.log 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41624.log 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42014.log.gz 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40765.log 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40102.log 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40921.log.gz 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40765.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41624.log.gz 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40648.log 2024-03-25T19:41:35.307 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42755.log 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40102.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40765.log.gz 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40414.log 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42755.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40102.log.gz 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40297.log 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40648.log.gz 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42677.log 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42755.log.gz 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40297.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40414.log.gz 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42677.log: gzip -5 --verbose 86.8% -- -- replaced with /var/log/ceph/ceph-client.admin.40297.log.gz /var/log/ceph/ceph-client.admin.42521.log 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42248.log 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42677.log.gz 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41311.log 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40180.log 2024-03-25T19:41:35.308 INFO:teuthology.orchestra.run.smithi019.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42521.log.gz 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41311.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42248.log.gz 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42287.log 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41311.log.gz 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- 86.0% /var/log/ceph/ceph-client.admin.40570.log 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40180.log.gz 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42716.log 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40570.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42287.log.gz 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40141.log 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42716.log: gzip -5 --verbose -- 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40570.log.gz 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr: /var/log/ceph/ceph-client.admin.40960.log 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40141.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39866.log 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42716.log.gz 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40141.log.gz 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40960.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:41:35.309 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36138.log/var/log/ceph/ceph-client.admin.39866.log: 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40960.log.gz 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41585.log 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39866.log.gz 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.36138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40258.log 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41585.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.36138.log.gz 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39788.log 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr:gzip 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41585.log.gz 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41467.log 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39788.log: /var/log/ceph/ceph-client.admin.40258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36332.log 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41467.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39788.log.gz 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40258.log.gz 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41467.log.gz 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.36332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40063.log 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:41:35.310 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36332.log.gz 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40336.log 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40063.log: /var/log/ceph/ceph-osd.2.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40063.log.gz 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40024.log 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40336.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40336.log.gz 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41116.log 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41389.log 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40024.log.gz 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40999.log 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41389.log: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41116.log.gz 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 86.3% --verbose -- /var/log/ceph/ceph-client.admin.42365.log 2024-03-25T19:41:35.311 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41389.log.gz 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41506.log 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42365.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40999.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42326.log 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41858.log 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41506.log: 87.2%/var/log/ceph/ceph-client.admin.42326.log: -- replaced with /var/log/ceph/ceph-client.admin.42365.log.gz 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42170.log 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42326.log.gz 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41858.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41506.log.gz 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38318.log 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42170.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41858.log.gz 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40609.log 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42053.log 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42170.log.gz 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38318.log: gzip/var/log/ceph/ceph-client.admin.40609.log: -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.42053.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40609.log.gz 2024-03-25T19:41:35.312 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38318.log.gz 2024-03-25T19:41:35.313 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42053.log.gz 2024-03-25T19:41:35.313 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41350.log 2024-03-25T19:41:35.313 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5/var/log/ceph/ceph-mon.c.log: --verbose -- /var/log/ceph/ceph-client.admin.41741.log 2024-03-25T19:41:35.313 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41350.log: /var/log/ceph/ceph-client.admin.41741.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41350.log.gz 2024-03-25T19:41:35.313 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41741.log.gz 2024-03-25T19:41:35.359 INFO:teuthology.orchestra.run.smithi188.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:41:36.116 INFO:teuthology.orchestra.run.smithi019.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:41:36.137 INFO:teuthology.orchestra.run.smithi188.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:41:36.138 INFO:teuthology.orchestra.run.smithi019.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:41:36.207 INFO:teuthology.orchestra.run.smithi188.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:41:36.389 INFO:teuthology.orchestra.run.smithi019.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:41:36.419 INFO:teuthology.orchestra.run.smithi019.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:41:36.472 INFO:teuthology.orchestra.run.smithi188.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:41:36.494 INFO:teuthology.orchestra.run.smithi188.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:41:40.278 INFO:teuthology.orchestra.run.smithi188.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:41:40.280 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-03-25T19:41:40.280 INFO:teuthology.orchestra.run.smithi188.stderr:real 0m5.017s 2024-03-25T19:41:40.280 INFO:teuthology.orchestra.run.smithi188.stderr:user 0m8.957s 2024-03-25T19:41:40.280 INFO:teuthology.orchestra.run.smithi188.stderr:sys 0m0.348s 2024-03-25T19:41:40.412 INFO:teuthology.orchestra.run.smithi019.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:41:40.554 INFO:teuthology.orchestra.run.smithi019.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:41:40.556 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-25T19:41:40.556 INFO:teuthology.orchestra.run.smithi019.stderr:real 0m5.327s 2024-03-25T19:41:40.556 INFO:teuthology.orchestra.run.smithi019.stderr:user 0m14.059s 2024-03-25T19:41:40.556 INFO:teuthology.orchestra.run.smithi019.stderr:sys 0m0.513s 2024-03-25T19:41:40.556 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:41:40.557 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310/remote/smithi019/log 2024-03-25T19:41:40.558 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:41:42.981 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310/remote/smithi188/log 2024-03-25T19:41:42.982 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:41:44.575 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:41:44.774 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ce6f2e409f824f58ba8b8b5e4a6c6dbf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:41:44.778 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:41:44.789 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:41:44.805 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:41:44.805 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:41:44.819 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:41:44.895 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:41:44.903 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:41:44.904 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:41:44.906 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:41:44.920 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:41:44.922 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:41:44.925 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:41:44.925 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-03-25T19:41:44.925 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 -273us[ -273us] +/- 68ms 2024-03-25T19:41:44.925 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -268us[ -271us] +/- 55ms 2024-03-25T19:41:44.925 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +801us[ +801us] +/- 90ms 2024-03-25T19:41:44.925 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:41:44.926 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:41:44.927 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-03-25T19:41:44.927 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 16 -259us[ -273us] +/- 68ms 2024-03-25T19:41:44.927 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -268us[ -282us] +/- 55ms 2024-03-25T19:41:44.927 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +810us[ +796us] +/- 90ms 2024-03-25T19:41:44.927 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:41:44.928 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:41:44.937 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:41:44.937 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:41:44.945 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:41:45.037 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:41:45.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:41:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:41:45.140 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:41:45.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:41:45.171 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:41:45.179 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:41:45.186 INFO:teuthology.task.internal:Duration was 6433.885056 seconds 2024-03-25T19:41:45.187 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:41:45.195 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:41:45.195 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:41:45.197 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:41:45.239 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:41:45.255 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:41:45.640 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:41:45.640 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2024-03-25T19:41:45.641 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:41:45.699 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-03-25T19:41:45.699 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:41:45.721 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:41:45.721 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:41:45.741 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:41:45.790 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:41:45.790 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:41:45.811 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:41:45.934 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:41:45.944 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:41:45.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:41:45.971 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:41:45.997 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:41:46.007 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:41:46.013 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:41:46.035 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2024-03-25T19:41:46.064 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-03-25T19:41:46.116 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:41:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:41:46.131 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:41:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:41:46.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:41:46.180 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:41:46.181 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310/remote/smithi019 2024-03-25T19:41:46.181 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:41:46.218 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310/remote/smithi188 2024-03-25T19:41:46.218 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:41:46.253 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:41:46.254 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:41:46.256 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:41:46.303 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:41:46.314 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:41:46.314 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:41:46.324 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:41:46.324 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:41:46.326 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:41:46.338 INFO:teuthology.orchestra.run.smithi019.stdout: 265672 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:41 /home/ubuntu/cephtest 2024-03-25T19:41:46.358 INFO:teuthology.orchestra.run.smithi188.stdout: 265583 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:41 /home/ubuntu/cephtest 2024-03-25T19:41:46.359 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:41:46.369 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:41:46.390 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:41:46.415 DEBUG:teuthology.nuke:shortname: smithi019 2024-03-25T19:41:46.415 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:41:46.437 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:11.947994', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIi43x2E99LNSF/3BhT5R/K4DVqzzb5IcZBBtUyCOAS2Q2zD3i8ZiImheSbixMlHx8DOIpUst3eNJrQHI7tnX70='} 2024-03-25T19:41:46.440 DEBUG:teuthology.nuke:shortname: smithi188 2024-03-25T19:41:46.440 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:41:46.466 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621310', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:11.949726', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0PwWWJMGrLoeOuDecx/Tzk2ixaznhC4GV2yeyjfgWbXxDkUGy7GYa1rGtCFqD+fTjGcat1iHyLvQ8iavV7mGw='} 2024-03-25T19:41:46.495 INFO:teuthology.orchestra.console.smithi019:Power off 2024-03-25T19:41:46.495 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:41:46.523 DEBUG:teuthology.orchestra.console.smithi019:power off output: Chassis Power Control: Down/Off 2024-03-25T19:41:46.523 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:46.538 INFO:teuthology.orchestra.console.smithi188:Power off 2024-03-25T19:41:46.538 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:41:46.548 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-03-25T19:41:46.561 DEBUG:teuthology.orchestra.console.smithi188:power off output: Chassis Power Control: Down/Off 2024-03-25T19:41:46.561 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:46.583 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-03-25T19:41:50.550 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:50.584 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:50.662 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-03-25T19:41:50.697 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-03-25T19:41:54.667 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:54.698 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:54.780 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-03-25T19:41:54.809 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-03-25T19:41:58.784 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:58.810 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:41:58.894 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is off 2024-03-25T19:41:58.919 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2024-03-25T19:41:58.996 INFO:teuthology.orchestra.console.smithi019:Power off completed 2024-03-25T19:41:59.020 INFO:teuthology.orchestra.console.smithi188:Power off completed 2024-03-25T19:41:59.124 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6433.885056018829 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ce6f2e409f824f58ba8b8b5e4a6c6dbf status: fail success: false 2024-03-25T19:41:59.125 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:41:59.207 INFO:teuthology.run:FAIL